SECURITY ADVISORY: web services (Aug. 11, 2011)

Aug. 11, 2011

The FreePBX development team has identified with some zero day security vulnerabilities related to httpd and php. These vulnerabilities may allow a remote user to gain full root control over a system, and are present in lots of popular asterisk-related distro’s.

The FreePBX development team strongly urges all user of the FreePBX Distro to immediately upgrade their systems and patch these vulnerabilities. Additionally, users are reminded never to keep their web port accessible to the internet.

To secure your system, please download the latest scripts found [url=http://www.freepbx.org/forum/freepbx-distro/distro-discussion-help/release-versions]here[/url]. Please remember that the upgrade scripts must be executed sequentially.

A big round of applause to my colleagues at [url=http://www.schmoozecom.com]Schmooze Com., Inc.[/url] for their tireless dedication to the community, for the sleepless nights they spent working on this (and many other!) issue, and for their swift response in releasing a patch to protect the users of the distro.

UPDATE: it seems this post has left a host of questions in its wake – please read the following replies to see if your questions have been answered yet, or reply with them if they havent been!

34 thoughts on “SECURITY ADVISORY: web services (Aug. 11, 2011)

  1. Guys at this time we do not have the specific CVE numbers and these could be older exploits. What was happening is we were getting reports in FreePBX Support of systems being hacked and were experiencing hackers going after our customers PBX’s and some of our core servers. The common thing we kept seeing was they were trying to get in through apache. We than had a customer report to us that they had a PCI Compliance scan done on there PBX and were told the following information.

    PHP- Remote attackers may be able to gain unauthorized access to the web server, cause a denial of service or information disclosure, or execute arbitrary code. Resolution PHP should be upgraded to 5.2.17 or higher for 5.2.x, to a version higher than 5.3.6 for 5.3.x

    APACHE- A remote attacker could crash the web server or execute arbitrary commands. Upgrade Apache 1.x to version 1.3.41-dev or higher, 2.0.x to version 2.0.64-dev or higher when available, or a version higher than 2.2.18

    Once we upgraded apache and php to the version they recommended the servers that were being attacked were no longer being attacked. Before upgrading apache and PHP if we killed the running exploits on a server they would spawn back up a hour later. Once we upgraded both packages they were no longer being spawned.

    This is all we know but for us it has stopped anyone that was being exploited. You will see that Apache in Centos is older than what is recommended here hence why we recommend that everyone upgrade.

  2. Hi Tony,

    Thanks for the update.

    Can you give more information as to how you were detecting these attacks? Was it something appearing in the Apache access logs?

    Many thanks, Matt

  3. Updates need to be done sequentially. This thread has a list and links to all the updates- http://www.freepbx.org/forum/freepbx-distro/distro-discussion-help/release-versions

    The updates are done using the root account on the machine or using terminal. The update is downloaded onto the FreePBX machine, made executable, and then run. I always use the tmp folder, but any directory will do. The line-by-line commands used for this latest (8/11/11) update would be:

    [code]wget http://upgrades.freepbxdistro.org/1.8.2.0/upgrade-1.8.2.0-2.sh
    chmod +x upgrade-1.8.2.0-2.sh
    ./upgrade-1.8.2.0-2.sh[/code]

    Again, these updates need to be run sequentially (in order). You can’t skip over a prior update.

    You can restart asterisk by running this:

    [code]amportal restart[/code]

  4. If you don’t know the actual bugs that need to be fixed, how did you fix them? How do you know the system is now secure? And what about those of us who use FreePBX on non-distro OS? Sorry but this “security advisory” is crap as it provides no useful advice whatsoever. This is a technical group; you need to provide some actual technical details.
  5. Ummm…

    [root@mysystem tmp]# cd /tmp
    [root@mysystem tmp]# wget http://upgrades.freepbxdistro.org/1.8.2.0/upgrade-1.8.2.0-2.sh
    –2011-08-11 14:24:14– http://upgrades.freepbxdistro.org/1.8.2.0/upgrade-1.8.2.0-2.sh
    Resolving upgrades.freepbxdistro.org… 199.102.239.6
    Connecting to upgrades.freepbxdistro.org|199.102.239.6|:80… connected.
    HTTP request sent, awaiting response… 200 OK
    Length: 20463 (20K) [application/x-sh]
    Saving to: `upgrade-1.8.2.0-2.sh’

    100%[== ….. ==>] 20,463 –.-K/s in 0.06s

    2011-08-11 14:24:14 (353 KB/s) – `upgrade-1.8.2.0-2.sh’ saved [20463/20463]

    [root@mysystem tmp]# chmod +x upgrade-1.8.2.0-2.sh
    [root@mysystem tmp]# ./upgrade-1.8.2.0-2.sh
    -bash: ./upgrade-1.8.2.0-2.sh: /bin/sh^M: bad interpreter: No such file or directory

    Downloaded and tried this twice, same result. 🙁

  6. Okay, I did this and it went fine until it got to Stage 3, then it got a bunch of PHP warning messages – see http://pastebin.com/Z4mY1hHw

    EDIT: I think I fixed the complaints by doing

    yum update php-mcrypt

    BUT I don’t know if the script completed properly or not due to the complaints – do I need to re-run it? Stages 1 and 2 seemed to go fine.

    EDIT #2: And where did php-mcrypt come from in the first place, I hear you ask, since it’s not part of the FreePBX distro? Well just a couple of days ago I [url=https://michigantelephone.wordpress.com/2011/08/09/problems-you-may-encounter-when-attempting-to-install-phpmyadmin-on-your-centos-server-and-how-to-solve-them/]installed phpMyAdmin[/url], because I think it’s a real handy tool to have, and I wanted to get a bunch of “garbage” records out of the CDR that had been generated by an endless loop I had inadvertently created (my own dumb mistake). And when I installed phpMyAdmin, it brought along libmcrypt and php-mcrypt as dependencies. I tried updating both, but only php-mcrypt actually had an update available today.

    When I do cat /etc/asterisk/freepbxdistro-version it seems to show that the update “took” so I guess I’m okay?

    # cat /etc/asterisk/freepbxdistro-version
    1.8.2.0-2

  7. Bill

    If I had more info I would provide it. We have no been able to figure out what the exact exploit was and have spent the past 5 days straight on it and even brought in some outside consultants. All we know is that once we upgrade to the latest apache the exploits die off as they can not get back into the box but if we downgrade back than we see them back in and doing port scans from the box to other boxes mainly trying to force there way into other peoples apache boxes or use there boxes for email relay.

    Believe me we are all very technical guys and are just as frustrated over this as the next guy.

    What do you propose next time we just keep out mouth shut and let others get exploited as we have seen in the forums has already happened. Some information is better than none don’t you agree.

  8. billsimon,
    Unfortunately, in the world of security vulnerabilities you can really never know if the NEXT vulnerability will affect you. As Tony mentioned:
    “Once we upgraded apache and php to the version they recommended the servers that were being attacked were no longer being attacked”
    and hence we feel that these upgrade are necessary for anyone that wants to protect agains the issue that we saw.

    Additionally, as Tony mentioned, anyone on other distors need only ensure that there running patched version of php and apache:
    [code]PHP should be upgraded to 5.2.17 or higher for 5.2.x, to a version higher than 5.3.6 for 5.3.x[/code]
    and
    [code]Upgrade Apache 1.x to version 1.3.41-dev or higher, 2.0.x to version 2.0.64-dev or higher when available, or a version higher than 2.2.18[/code]

    Finally, we have always stressed that FreePBX is built to be a front end to asterisk – useful for a sysadmin or a group of sysadmin. It was not intended to be a public facing web app/site. Hence, we always recommend to user that the web portion is NOT ACCESSIBLE from the internet. While we take FreePBX’s security seriously, this vulnerability is way beyond the scope of FreePBX (or anything that we could patch/fix at the FreePBX level) and should help to convince users to keep their boxes off the net at all costs.

  9. Hi mbrevda,

    I’m sure everyone appreciates the warning but the problem is there is not enough information to work out what the compromise is. I know you say you don’t have enough information but some more clues would be good.

    For instance, you say you saw lot of boxes that had root compromised. I’m assuming that you did not just upgrade Apache/PHP on these same boxes and the problem was cured?

    So I’m assuming you updated Apache and PHP on others boxes, and have not seen these ones compromised. But how do you know they just haven’t been compromised *yet*? If you don’t know what the attack vector is how are you detecting it?

    If this was a general issue affecting the current released version of Apache and/or PHP with CentOS then this would be a widespread issue, widely reported surely? Not just with FreePBX.

    So isn’t it more likely that it’s part of FreePBX being compromised?

    Also, do you know for sure the root account was compromised? You don’t need root access to scan for other servers to attack.

    Either way more information about what you were seeing in the Apache logs, what form the attacks took, and how to detect if your servers are being scanned would definitely be welcome.

    Thanks, Matt

  10. We would ps awx and see a ton of process called exim running which was executing some perl script and most of the time you could see a SCREEN session running.

    If you did a screen -x you could watch the port scan running. In each example we would see they were running scans of IP’s for /phpmyadmin or /myadmin. At other times they were doing scans to find email SMTP relay servers to exploit from other peoples boxes.

    Most of these scripts were being fed from other hacked boxes running IRC servers to command and control the IP addresses the scripts should be executing.

    Most of the time the scripts were being save in /usr/game/

    Hope this helps everyone.

  11. [quote]So I’m assuming you updated Apache and PHP on others boxes, and have not seen these ones compromised. But how do you know they just haven’t been compromised *yet*? If you don’t know what the attack vector is how are you detecting it?[/quote]
    That was my point above – you can never know 1000% for sure. In this case it definitely helped with at least some issues. Are you a million precent safe after you apply the upgrades? Only time will tell. Unfortunately, we do not have the resources to employ the cream of the crop in security people to try to detect every posible theoretical issue.
    [quote]
    If this was a general issue affecting the current released version of Apache and/or PHP with CentOS then this would be a widespread issue, widely reported surely? Not just with FreePBX.
    [/quote]
    indeed.
    [quote]
    So isn’t it more likely that it’s part of FreePBX being compromised?
    [/quote]
    No, this issue was also seen on systems that weren’t running FreePBX at all
    [quote]
    Also, do you know for sure the root account was compromised? You don’t need root access to scan for other servers to attack.[/quote]
    No. However the hacks were extremely sophisticated (as described by the security consultants we brought it in) and we have reason to suspect “they” may have had root-level access
  12. Yes, I will agree that some information is better than none. Thank you. Have you gotten any response from Apache on this issue? Attempting to research it is frustrating. Google blocks the search for “apache exploit” in quotes, claiming no results. (Really?) Apache’s change log for httpd from 2.2.18 to .19 shows only one patch:

    [quote] -*- coding: utf-8 -*-
    Changes with Apache 2.2.19

    *) Revert ABI breakage in 2.2.18 caused by the function signature change
    of ap_unescape_url_keep2f(). This release restores the signature from
    2.2.17 and prior, and introduces ap_unescape_url_keep2f_ex().
    [Eric Covener]
    [/quote]

    If 2.2.18 has the vulnerability and 2.2.19 does not, then more was done than was written here in the log.

  13. Yes I would love to know more also but after investing 5 grand of money into finding the root exploit and not getting anywhere we gave up as we just do not want to keep spending money on the issue since the upgrade resolved it for us.

    All I can say is when we killed the running processes on a box they would spawn back up within 30 minutes. After upgrading apache and php as outlined above and killing the processes they never started back up on any boxes that we did this on. That was enough for us to close out our 5 days of hell and move back to productive work.

  14. Okay, this is upsetting…

    I have pretty strong feelings about doing upgrades that take over your system. Basically, it’s my feeling that it’s MY system, not YOURS, even if I am running your software. One of the reasons I’ve declined to use a competing distro is because I feel they take just a little too much control away from a system administrator.

    I’m not sure of everything this upgrade did, but it definitely did one thing I don’t like, and may have done another (I can’t be sure). The thing I’m not sure about is that now ALL FreePBX modules are enabled, and I seem to recall that after the original distro install two or three were disabled by default. This isn’t a big deal as long as the newly-enabled modules (if any) don’t screw up anything on my system.

    But the thing that more upsets me is that I happened to go into /etc/yum.repos.d/ and noticed that all repos except the FreePBX repo were GONE – I mean they were totally deleted! I personally only had two additional repos — the dag repo and the webmin repo — but some may have taken the trouble to install others.

    Now, i understand that during an upgrade you may not want to be pulling software from a repo you can’t trust. Fair enough, but it’s overstepping your bounds to simply delete any other repos that someone may have installed. What the script SHOULD do is move them to a safe place, and then at the end of the script, move them back. If you want, you can even print a warning that there are repos that you think shouldn’t be there, and ask the system administrator to confirm that they are really wanted. But ultimately, you shouldn’t be deleting anything that the administrator may have taken the trouble to install on a system, because down the road he or some automated process might assume it’s still there. I can see why you might not like the dag repo being enabled during an upgrade, but I can’t see how the webmin repo would cause you any issues, and Webmin probably depends on it to do software and module upgrades.

    Please don’t turn into that “other” distro that treats us like we’re all children!

    By the way, is there anything else you delete or change during the update process that we might want to know about (besides actual software updates, of course)?

  15. You can simple look at the upgrade script and see what it does. We do not compile or hide what is in our update scripts. We always recommend that you look at upgrade scripts regardless what platform you are using them from.
  16. tonyclewis, I’m not a programmer. And I’m just saying, there are certain boundaries you shouldn’t cross, like not permanently deleting software (or repositories) that a user may have installed.

    Would it really be that hard to be considerate and copy anything you don’t want present to a safe place, then restore it after the fact? Even if I can figure out what you are doing in your scripts (which might not always be that easy), there are doubtless users that are far less technically astute, that could look at your script all day and not really understand what you’re doing. I’ll grant that such users are probably less likely to be installing additional software or repos in the first place, but just because it’s less likely doesn’t mean it’s impossible. They might have followed “cookbook” style instructions from a web site or blog to install a particular program (such as Webmin, which is very popular with users such as myself that don’t care much for the Linux command line).

  17. And to answer your question all we do is a update all for modules not install any modules that have not been installed already and yes we clear out all other repos to make sure there are not conflicts. I can look at just moving them and moving them back after the upgrade for the future as that did not occur to me but once again it is all in plain english to see and we spit out comment as the upgrade runs like “Replace repos with only FreePBX Distro since some people have added other repos which can break updates”
  18. tonyclewis, what I was really asking was, do you by any chance set any modules to “enabled” that were previously present on the system but disabled? I know you didn’t install anything that wasn’t already installed.

    Anyway, thank you for giving this some consideration. When a script is running sometimes things scroll by pretty quickly. Looking at the script, I now see where that’s in there, but didn’t really consciously notice it as it scrolled by (on the other hand, I DID look in that directory, so maybe I noticed it subconsciously or something). At the end I was a bit more concerned about those PHP warning messages and trying to figure out what might have caused them, so that was what pretty much occupied my attention for several minutes.

    Thanks for the explanation, I now feel more reassured! And THANK YOU for helping keep our systems secure, THAT is very much appreciated. The thing about the disappearing repos was really one of only a VERY few things I’ve found in the FreePBX Distro that I would consider a negative — overall, installing and using the distro has been a pretty smooth experience, given that it’s relatively new and a few of the kinks are still being worked out. On the other hand, I wasn’t even aware until today that you guys had upgrade scripts, so that is good to know!

  19. What about PHP version 5.1.6? Does it need upgrading?

    And the FreePBX servers we have are exposed over internet via HTTPS (custom certificates) – but not via 443 port, but a random redirected port, is this HTTPS with SSL also a problem?

  20. Ok so we just had a client in paid support inform us they thought they were hacked. We logged in and saw in the ps aux the following

    root 12379 3.4 0.2 49960 10984 pts/0 S+ 13:25 0:00 /usr/bin/perl ./mycnf 174.142.177.241 10000 0
    root 12380 3.2 0.2 49960 10984 pts/0 S+ 13:25 0:00 /usr/bin/perl ./mycnf 174.142.177.242 10000 0
    root 12381 3.3 0.2 49960 10980 pts/0 S+ 13:25 0:00 /usr/bin/perl ./mycnf 174.142.177.243 10000 0
    root 12382 3.4 0.2 49960 10984 pts/0 S+ 13:25 0:00 /usr/bin/perl ./mycnf 174.142.177.244 10000 0

    There were about 2000 lines of that and the IP’s kept changing as the port scan was running. Load average was up to 150 in TOP.

    The file this time was located in /usr/books/.n I tar’d up the files and if anyone wants them please PM me with your email and I will forward them to you.

    The only ports open on the firewall for this customer was 80, 5060 and 10000-20000 for SIP. Everything else was closed down. I see nothing in the access files for apache on someone failing to log in and fail2ban did not detect any failed logins. This was not a FreePBX Distro but a different FreePBX based system. They were running an older apache and php and advised them to upgrade those packages but the customer decided to do a re install instead and than upgrade the packages.

    If anyone else runs into this please be very careful killing the processes as each time we have done this they than DoS attack the IP that you killed the process from.

  21. And here is another one from the same customer different box.

    root 10209 2.0 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
    root 10210 2.0 0.0 1740 384 pts/0 R+ 15:20 0:00 ./exim new vuln
    root 10211 1.8 0.0 1740 384 pts/0 R+ 15:20 0:00 ./exim new vuln
    root 10212 2.0 0.0 1740 384 pts/0 R+ 15:20 0:00 ./exim new vuln
    root 10213 2.0 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
    root 10214 1.8 0.0 1740 384 pts/0 R+ 15:20 0:00 ./exim new vuln
    root 10215 0.2 0.1 7948 2060 ? Ss 15:20 0:00 sshd: root@pts/
    root 10217 2.2 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
    root 10218 2.2 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
    root 10219 1.8 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
    root 10220 1.8 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
    root 10221 2.0 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
    root 10222 2.0 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
    root 10223 1.7 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
    root 10224 1.7 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
    root 10225 1.7 0.0 1740 384 pts/0 R+ 15:20 0:00 ./exim new vuln
    root 10226 1.7 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
    root 10227 1.7 0.0 1740 384 pts/0 R+ 15:20 0:00 ./exim new vuln
    root 10228 1.6 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
    root 10229 1.6 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
    root 10230 0.5 0.1 4752 1492 pts/2 Ss 15:20 0:00 -bash
    root 10254 1.5 0.0 1740 384 pts/0 R+ 15:20 0:00 ./exim new vuln
    root 10255 1.5 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
    root 10256 1.5 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
    root 10257 1.0 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
    root 10258 1.5 0.0 1740 384 pts/0 R+ 15:20 0:00 ./exim new vuln
    root 10259 1.0 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
    root 10260 1.5 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
    root 10261 1.5 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
    root 10262 1.0 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
    root 10263 1.5 0.0 1740 384 pts/0 R+ 15:20 0:00 ./exim new vuln
    root 10264 1.5 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
    root 10265 1.5 0.0 1740 384 pts/0 R+ 15:20 0:00 ./exim new vuln
    root 10266 1.0 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
    root 10267 1.0 0.0 1740 384 pts/0 R+ 15:20 0:00 ./exim new vuln
    root 10268 1.0 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
    root 10269 1.0 0.0 1740 384 pts/0 R+ 15:20 0:00 ./exim new vuln
    root 10270 1.0 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
    root 10271 1.0 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
    root 10272 1.0 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
    root 10273 1.0 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
    root 10274 1.0 0.0 1740 384 pts/0 S+ 15:20 0:00 ./exim new vuln
    root 11773 0.0 0.0 1748 412 pts/0 S+ 14:20 0:00 ./exim new vuln
    root 11814 0.0 0.0 4564 1064 pts/0 S+ 14:20 0:00 sh -c perl x.pl
    root 11815 0.0 2.6 38448 34564 pts/0 S+ 14:20 0:00 perl x.pl 202.1
    root 12314 0.0 0.0 1748 412 pts/0 S+ 14:21 0:00 ./exim new vuln
    root 12377 0.0 0.0 4564 1068 pts/0 S+ 14:21 0:00 sh -c perl x.pl
    root 12378 0.0 1.8 28208 24328 pts/0 S+ 14:21 0:00 perl x.pl 202.1
    nagios 18092 0.0 0.0 5080 944 ? Ss 12:05 0:00 /usr/local/nagi
    root 18899 0.0 0.0 1748 412 pts/0 S+ 14:29 0:00 ./exim new vuln
    root 18953 0.0 0.0 4564 1068 pts/0 S+ 14:29 0:00 sh -c perl x.pl
    root 18954 0.0 8.1 110128 106244 pts/0 S+ 14:29 0:00 perl x.pl 202.1
    root 23242 0.0 0.0 5032 1156 ? Ss 13:28 0:02 SCREEN
    root 23243 0.0 0.1 4752 1528 pts/0 Ss 13:28 0:00 /bin/bash
    root 23261 0.0 0.0 4756 860 pts/0 S+ 13:28 0:00 /bin/bash

    Notice the screen session. If you go into screen you will see the following.
    [+] 202.161.45.177
    [+] 202.162.217.59
    [+] 202.162.217.57
    [+] 202.162.217.56
    [+] 202.162.217.58
    [+] 202.162.217.60
    [+] 202.162.217.61
    [+] 202.162.77.52
    sh: fork: Cannot allocate memory
    [+] 202.162.78.9
    [+] 202.162.77.36
    sh: fork: Cannot allocate memory
    [+] 202.162.78.8
    sh: fork: Cannot allocate memory
    [+] 202.162.77.25
    sh: fork: Cannot allocate memory
    [+] 202.162.78.59
    sh: fork: Cannot allocate memory
    [+] 202.164.172.195
    [+] 202.164.172.197
    sh: fork: Cannot allocate memory
    [+] 202.164.189.226
    [+] 202.164.189.225
    [+] 202.164.189.228
    sh: error while loading shared libraries: libtermcap.so.2: failed to map segment from shared object: Cannot allocate memory
    sh: fork: Cannot allocate memory
    [+] 202.165.247.91
    [+] 202.167.231.47
    [+] 202.167.231.46
    [+] 202.168.64.111
    [+] 202.168.64.110
    [+] 202.170.120.224
    [+] 202.170.120.225
    [+] 202.170.122.83
    [+] 202.170.126.52
    [+] 202.170.127.103
    [+] 202.170.127.76
    [+] 202.170.127.66
    [+] 202.170.127.73
    [+] 202.170.127.9
    [+] 202.170.127.84
    [+] 202.170.38.231
    [+] 202.171.133.31
    [+] 202.171.136.198
    [+] 202.171.136.195
    [+] 202.171.136.205
    [+] 202.171.136.214
    [+] 202.171.136.217
    [+] 202.172.169.16
    [+] 202.172.169.42
    [+] 202.172.169.18
    [+] 202.172.32.144
    [+] 202.172.32.223
    [+] 202.173.130.5
    [+] 202.173.138.29
    [+] 202.173.141.190
    [+] 202.173.142.160
    [+] 202.173.140.192
    [+] 202.173.151.16
    [+] 202.173.180.34

    I tar’d up the files this time from /usr/books if anyone wants them.

  22. That machine is totally compromised. It’s a goner.

    They’re exploiting exim – the mail program. Check out this slashdot article: http://it.slashdot.org/story/10/12/1…it-In-the-Wild

    Bite the bullet and take that machine off line immediately!

    You can just shut down the outward facing eth0 or eth1 if you want to do a forensic examination the system while sitting in front of the machine.

    The hacker is using a file called “x.pl” to do nasty things. Find it and delete it. That will clean up the stuff you can see.

    If it were me, I’d format the disk and reinstall the most recent version of the OS.

  23. I’ve tried to collect all of what we currently know about this vulnerability in one place.
    Moderator note: No need to go off to other site, we will keep the information available here. Thanks.

    Here’s a summary of what we currently know about this vulnerability:

    [b]Vulnerability:[/b] Any stock CentOS 5.x or 6.x system with Apache and PHP exposed to Internet access with no IP address restrictions (WhiteList).

    [b]How Do They Get In[/b]: Some (perhaps unknown) vulnerability in stock versions of Apache and PHP on CentOS systems allows the attacker to gain system access. We really don’t know any more than that at this juncture. But this does not appear to be a PHPmyAdmin exploit as that utility is locked down by secure htaccess password on some systems that have been compromised. Fail2Ban is not detecting hack attempts so it appears the attacker is walking right in with this exploit.

    [b]Privileges[/b]: Still unclear whether attacker is gaining root access or merely same access as enjoyed by Apache on the attacked system. To do what they’re doing would NOT require root privileges on your system. The attacker brings a customized version of WebMin with their own password.

    [b]What Happens Once They’re In:[/b] In a nutshell, your system is turned into a zombie. Using perl and WebMin (their own version), they can interconnect your server into a worldwide network of machines used to launch denial-of-service and other malicious attacks against other systems on the Internet.

    [b]How Do I Know If My Machine Has Been Compromised?[/b] Examine some of the previous comments in this thread. Run ps awx on your server and look for long lists of processes running perl scripts. Look in the /usr directory for a directory called game, games, books, etc. Inside those directories, run ls -all which will show hidden files beginning with a period. There will be a directory called .n or .s or something similar. Look in /etc/cron.daily. There will be a new script as outlined in this thread. NOTE: The zombie software is old and signatures already exist in anti-virus programs. The exploit to gain access may be entirely new.

    [b]What Should Be in /usr[/b]? On a stock FreePBX Distro system, you should see the following directories:

    bin etc games include java kerberos lib libexec local sbin share src tmp X11R6

    The games directory will be empty when you ls -all

    [b]What Should Be in /etc/cron.daily?[/b] On a stock FreePBX Distro system, you should see the following files:

    0anacron 0logwatch cups logrotate makewhatis.cron mlocate.cron prelink rpm tmpwatch

    [b]How to Fix It:[/b] If your system has been compromised, reformat the disk and reinstall. If they haven’t gotten in or if you’ve started over, (1) immediately turn off Internet access to web services on your servers. You can implement a whitelist of safe IP addresses for web access using IPtables or your hardware-based firewall. (2) Upgrade Apache to latest version and PHP to latest 5.2 or 5.3 version.

    NOTE: Just because your system has not yet been compromised does NOT mean you are safe. Your system still needs to be secured. Turn OFF Web Access Now!

    [B]For Standard PIAF servers and Proxmox PIAF installs with IPtables:[/B]

    [B]How to Temporarily Disable Web Access[/B]: Log in as root. Issue command: [B]iptables -D INPUT -p tcp -m tcp –dport 80 -j ACCEPT[/B]

    [B]How to Temporarily Enable Web Access[/B]: Log in as root. Issue command: [B]iptables -A INPUT -p tcp -m tcp –dport 80 -j ACCEPT[/B]

    [B]How to Temporarily Enable Web Access for Specific IP Address [/B]: [B]iptables -A INPUT -p tcp -m tcp -s 123.45.67.8 –dport 80 -j ACCEPT[/B]

  24. To start I`m very new to linux and asterix and FreePBX, I downloaded the disro and installed on a old PC, bought a linksys PAP2 ATA and a Linksys SPA3000.
    Followed how-to`s and easly got 2 EXTNs on the PAP2, 1 on my laptop using a softphone, setup the SPA3000 to make and recive PSTN call, and a trunk form Sipgate, All working great.

    After following the instruction above I can no longer login to “Free BPX Administrator” the system is running and calls are routing.

    Can anyone help me get back into the Web admin page, it dose popup the login dialoge but then sits on a blank page.!!? grr

  25. Hi scottmoss. Perhaps the wrong place to post this, but I’ll respond so some of the developers will see this. Before 2.9 you could edit /etc/amportal.conf The parameter authtype could be changed from =database to =none and after an amportal restart you could get into the GUI with no password and reset you password. Then set it back to database and another amportal restart you you were back in business. Now amportal.conf says don’t edit it. So I don’t know where to change this, or if it is still possible to gain access if you have ssh or console access.

    Any word from the higher ups? Thanks

  26. Can you install a wireshark in front of a reinstalled server to see the attack attempt? [might be your 5 grand answer]

    Is it wise to schedule a daily yum -y update? [we need automatic updates]

    How about install a FreePBX onto a public IP with no firewall. First, package FreePBX into an OpenVZ container ( wink, wink ). Then host it on a $10 VPS. Login every day to take a system snapshot (recursive file listing, hidden files, file sizes, ps auxw) and cron an email to yourselves which shows you diff from the day before. [maintain a honeypot for us]

Leave a Reply