I haven't heard that one before, something is hanging those processes - how much memory do you have on your system?

BTW, I submitted a paid ticket with log files if that would help... ZPV-610725

Thanks

escape2mtns,

if you submitted a paid ticket then that team will have a look and work with you as needed. The fact that dialparties is hanging is troublesome, there may be something with your php.ini but best to leave that discussion with the engineer who is going to help you.

no - it's probably an undefined variable in recordingcheck that may need to be fixed, but it would not be related to what you are seeing.

Looking through the log files I noticed the following on multiple occasions:

app_queue.c: The device state of this queue member, Local/200@from-internal/n, is still 'Not in Use' when it probably should not be! Please check UPGRADE.txt for correct configuration settings.

[Jul 7 10:39:29] VERBOSE[27346] logger.c: dialparties.agi: Methodology of ring is 'none'
[Jul 7 10:39:29] VERBOSE[27346] logger.c: -- dialparties.agi: Added extension 200 to extension map
[Jul 7 10:39:29] VERBOSE[27346] logger.c: -- dialparties.agi: Extension 200 cf is disabled
[Jul 7 10:39:29] VERBOSE[27346] logger.c: -- dialparties.agi: Extension 200 do not disturb is disabled
[Jul 7 10:39:29] VERBOSE[27346] logger.c: dialparties.agi: Extension 200 has ExtensionState: 0
[Jul 7 10:39:29] VERBOSE[27346] logger.c: -- dialparties.agi: Checking CW and CFB status for extension 200
[Jul 7 10:39:29] VERBOSE[27346] logger.c: -- dialparties.agi: dbset CALLTRACE/200 to 6514427031
[Jul 7 10:39:29] VERBOSE[27346] logger.c: -- dialparties.agi: Filtered ARG3: 200
[Jul 7 10:39:29] VERBOSE[27353] logger.c: == Manager 'admin' logged off from 127.0.0.1
[Jul 7 10:39:29] VERBOSE[27346] logger.c: -- AGI Script dialparties.agi completed, returning 0
[Jul 7 10:39:29] DEBUG[27346] app_macro.c: Executed application: AGI
[Jul 7 10:39:29] VERBOSE[27346] logger.c: -- Executing [s@macro-dial:7] Dial("Local/200@from-internal-1eb8,2", "SIP/200||trM(auto-blkvm)") in new stack
[Jul 7 10:39:29] VERBOSE[27346] logger.c: -- Called 200
[Jul 7 10:39:29] VERBOSE[27343] logger.c: -- Local/200@from-internal-1eb8,1 is ringing
[Jul 7 10:39:30] VERBOSE[27346] logger.c: -- SIP/200-082f0f98 is ringing
[Jul 7 10:39:31] VERBOSE[27346] logger.c: -- SIP/200-082f0f98 answered Local/200@from-internal-1eb8,2
[Jul 7 10:39:31] VERBOSE[27346] logger.c: -- Executing [s@macro-auto-blkvm:1] Set("SIP/200-082f0f98", "__MACRO_RESULT=") in new stack
[Jul 7 10:39:31] DEBUG[27346] app_macro.c: Executed application: Set
[Jul 7 10:39:31] VERBOSE[27346] logger.c: -- Executing [s@macro-auto-blkvm:2] DBdel("SIP/200-082f0f98", "BLKVM/191/SIP/SIP1-b6d083a8") in new stack
[Jul 7 10:39:31] VERBOSE[27346] logger.c: -- DBdel: family=BLKVM, key=191/SIP/SIP1-b6d083a8
[Jul 7 10:39:31] DEBUG[27346] app_macro.c: Executed application: dbDel
[Jul 7 10:39:31] DEBUG[27346] app_dial.c: Macro exited with status 0
[Jul 7 10:39:31] DEBUG[27343] app_queue.c: Dunno what to do with control type -1
[Jul 7 10:39:31] VERBOSE[27343] logger.c: -- Local/200@from-internal-1eb8,1 answered SIP/SIP1-b6d083a8
[Jul 7 10:39:31] VERBOSE[27343] logger.c: -- Stopped music on hold on SIP/SIP1-b6d083a8
[Jul 7 10:39:31] WARNING[27343] app_queue.c: The device state of this queue member, Local/200@from-internal/n, is still 'Not in Use' when it probably should not be! Please check UPGRADE.txt for correct configuration settings.

Could this be related? I looked in the UPGRADE.txt file and I don't see anything.

In the UPGRADE.txt file I did see this:

* The exit behavior of the AGI applications has changed. Previously, when
a connection to an AGI server failed, the application would cause the channel
to immediately stop dialplan execution and hangup. Now, the only time that
the AGI applications will cause the channel to stop dialplan execution is
when the channel itself requests hangup. The AGI applications now set an
AGISTATUS variable which will allow you to find out whether running the AGI
was successful or not.

Previously, there was no way to handle the case where Asterisk was unable to
locally execute an AGI script for some reason. In this case, dialplan
execution will continue as it did before, but the AGISTATUS variable will be
set to "FAILURE".

A locally executed AGI script can now exit with a non-zero exit code and this
failure will be detected by Asterisk. If an AGI script exits with a non-zero
exit code, the AGISTATUS variable will be set to "FAILURE" as opposed to
"SUCCESS".

Could the new handling for AGI scripts be causing the dialparties to hang? I did see many log entries where the scripts exited with non-zero.

Thanks

It's probably a coincidence but we're all 3 running on HP DL360/DL380's.

We've dropped all our queues down to 5 or fewer static agents to see how it goes today.

Sounds extremely logical to me. The problem I have is that we have changes to the Queues on a regular basis from FPBX. Where could we "fix" this in the scripts that make the modification in the queues_additional.conf file?

Is FPBX develpoment considering this change for call center solutions like ours?

Thanks for the insight.

I'd love to take advantage of all the new stuff when they get it working but I need to get to a point where we're not disconnecting callers tomorrow. I don't have a bullet-proof vest and I'm not sure how to get by the call center director and live to tell about it. I've got a family to think about. :)

The FreePBX engineer I spoke with today said that he thought this sounded like a reasonable solution. We discussed the fact that I won't have asterisk-based DND, Call Forward, etc. but we're not using those anyway. This is a call center environment where the agents put the phone devices on DND when they're away and we don't use call forwarding.

What are the other implications? All I can go on is recommendations from the experts on how to fix this and right now I'm hearing that "in the field" this is how it's fixed.

I'm open to anything...

Thanks

I have just tested and it works for me.

What I did was change "Local" to "SIP" and deleted out the "from-internal" in the "/var/www/html/admin/modules/queues/page.queues.php" and then saved changes to the queues from within FPBX. This successfully changed Local to SIP in the queue_additional.conf like escape2mtns describes above.

This is working well at the moment I will observe tomorrow under high traffic conditions how this works out.

Philippe,
thank you for your input because I was sure that there were implications....but at the moment I need a solution for the dialparties.agi script hanging every day and going to 180% load on my server.

What main implications does this change to SIP have? ...because since I have done it my server has gone down to 20% CPU usage and I have perfect voice quality on all calls which is my first objective obviously seeing that this is a 100 agent call center.

What is you suggestion for someone like me that is not very savvy on programming etc? I have implemented 1.4.26RC5 seeing that you had mentiond something having been implemented in 1.4.25 or 26 but still today I was having the same issues.

How can I get hold of the other solution you mention with the "hints"?

Thanks for your input.

At the end of the first full business day using SIP queue members, we've now gone over 24 hrs without a crash/restart or hung dialparties.agi (for the first time).

System uptime: 1 day, 2 hours, 37 minutes, 59 seconds
CPU: 0%

Everything else worked as expected. So far so good.

I am happy as a pig in sh.t. The call center has been stable for 3 days now using SIP instead of Local. I wonder when the Asterisk team finds out what they broke in the way the AGI are handled. I`m no expert but I`d say there is something gravely wrong somewhere there.

Here is the change I made in:
/var/www/html/admin/modules/queues/page.queues.php

Changed from: (just serach for "Local")
$members[$key] = "Local/".$members[$key]."@from-internal/n,".$penalty_val;
to:
$members[$key] = "SIP/".$members[$key].",".$penalty_val;

This is obviously a temporary fix until the queues module is updated by FPBX again and at some point in time I hope there is a solution for the dialparties hanging issue to be able to revert back to the original way of handling these calls as intended by FPBX.

Actually after my last post, I think the bug IS in FreePBX, and I've reported that here:

http://www.freepbx.org/trac/ticket/3792#preview

along with the fix that is needed in dialparties.agi

a simple one line fix that definitely works is to change

$astman = new AGI_AsteriskManager( );

to

$astman = new AGI_AsteriskManager( );
$astman->pagi = & $AGI;

You may be able to change the line to

$astman = $AGI->new_AsteriskManager();

instead. That does seem to work for me, but it does change some other things so I'm not as confident about it

I'm using Asterisk 1.6.0.9

The (current?) FreePBX 2.5.1 download doesn't have code as you describe, and the 'is_ext_avail' function uses the manager connection, so that's what I've been working with.

Using the original dialparties.agi, an incoming call to a single extension would have a second or two delay before the internal extension started ringing, even though the outside line would give the ring tone immediately. With my suggested fix, the inside line starts ringing almost immediately. Queues are similar, with them ringing immediately, and no excessive CPU usage, delays or failure.

I don't know if you changed the code to what you described to try and fix this delay problem (because it would), but using the AGI->new_AsteriskManager() function instead of 'new AGI_AsteriskManager()' does fix it, and seems a simpler change, and it will work with Asterisk 1.4 as well :)

I'm not sure why the PHP 'error_log' function is so slow, but if that is different speeds on different computers (eg because of different PHP configurations), that would explain why some people have problems, and other people can't reproduce it.

I'm also not sure why phpagi uses error_log if the AGI_AsteriskManager isn't linked to an AGI instance, so maybe that should be reported as a bug in phpagi as well.

Let's examine this, I'm going to 'think outloud' here to see if we can get to the bottom of this. Here are the only differences, to the best that I can tell (and I have only had 1 coffee so ...).

Firs, this is influenced by my current phpagi.conf file which includes:

[festival]
text2wave=/usr/src/festival/bin/text2wave
tempdir=/var/lib/asterisk/sounds/tmp/
[asmanager]
server=localhost
port=5038
username=admin
secret=amp111

The config setting of the astman handle as is today are:

ASM config: Array
(
    [festival] => Array
        (
            [text2wave] => /usr/src/festival/bin/text2wave
            [tempdir] => /var/lib/asterisk/sounds/tmp/
        )

    [asmanager] => Array
        (
            [server] => localhost
            [port] => 5038
            [username] => admin
            [secret] => amp111
        )

)

Then with your change of:

<?php
$astman = $AGI->new_AsteriskManager();

ASM config: Array
(
    [festival] => Array
        (
            [text2wave] => /usr/src/festival/bin/text2wave
            [tempdir] => /var/lib/asterisk/sounds/tmp/
        )

    [asmanager] => Array
        (
            [server] => localhost
            [port] => 5038
            [username] => admin
            [secret] => amp111
            [festival] => Array
                (
                    [text2wave] => /usr/src/festival/bin/text2wave
                    [tempdir] => /var/lib/asterisk/sounds/tmp/
                )

            [asmanager] => Array
                (
                    [server] => localhost
                    [port] => 5038
                    [username] => admin
                    [secret] => amp111
                )

            [phpagi] => Array
                (
                    [error_handler] => 1
                    [debug] =>
                    [admin] =>
                    [tempdir] => /var/spool/asterisk//tmp/
                )

        )

)

Looking through the code, I'm not seeing the above config differences having an effect. The only difference between the way it is now and your change is the config that is passed into the astman handle resulting from:

<?php
$this->asm->pagi =& $this;
$this->config =& $this->asm->config;

Those change that you pointed out mean that with the new call, the asm->pagi is set telling the astman handle that it has access to AGI, so to speak (e.g. running in an AGI session) So ... this looks like it may be the crux of the difference, because the other configs above don't look like they make a difference for AMI, but his function:

<?php
    function log($message, $level=1)
    {
      if($this->pagi != false)
        $this->pagi->conlog($message, $level);
      else
        error_log(date('r') . ' - ' . $message);
    }

Which means it writes to error_log if you have not set pagi (as is the case today) but to conlog() if you have. The conlog function is:

<?php
    function conlog($str, $vbl=1)
    {
      static $busy = false;

      if($this->config['phpagi']['debug'] != false)
      {
        if(!$busy) // no conlogs inside conlog!!!
        {
          $busy = true;
          $this->verbose($str, $vbl);
          $busy = false;
        }
    }
}

Since in my case, config['phpagi']['debug'] is false, nothing gets written. And if a lot was otherwise getting written, that could very well bog things down (though it's strange that writing out log messages would have such a dramatic effect.

Now be aware that the PHPAGI Config module, if being used, has settings for debug and error which can further influence the behavior. (If you are seeing the debug message as verbose messages, then you probably want to turn that level off as well to reduce the noise).

So ... I think you may be right in that we want to change the code, though will probably only do it initially in 2.6 since we always approach such fundamental changes cautiously, but seems like it should be:

===================================================================
--- dialparties.agi     (revision 7882)
+++ dialparties.agi     (working copy)
@@ -62,7 +62,7 @@
 // If we are 1.6 then we have the EXTENSION_STATE() function and don't need to use the manager
 //
 if (!$has_extension_state) {
-       $astman = new AGI_AsteriskManager( );
+       $astman = $AGI->new_AsteriskManager();
        if (!$astman->connect("127.0.0.1", $ampmgruser , $ampmgrpass)) {
                exit (1);
        }

Question to Phillippe and PSCS,
what changes should I make for testing? Is it sufficient only changing the dialparties.agi or do I need to take out the logging feature as well??

At the mome nt I am still using the SIP instead of Local but as Phillippe mentions there are some drawbacks to this such as the transfer of calls issue so I would like to get back to a working solution with "Local" but without the long wait times eg in transfers etc.

Appreciate your response here.

warlock67,

the change has already been checked into 2.6 #3792. If you test on 2.5 and have some solid results it will help to drive us to back port it to 2.5. My general approach is to introduce it cautiously, even though it looks pretty straight forward per the long post above. Further testing will help convince us that it should go into 2.5 (and maybe 2.4 also).

I do still plan on getting a tarball of 2.6 beta out within the next week I hope as well, but that will be beta though pretty solid.

warlock67,

if you could make sure to ping the ticket in the next day or two once you are pretty certain this is solid, it will trigger me to look at merging that fix into 2.5, it is sounding pretty convincing to me.