Heavy Queue Usage in FreePBX

When it comes to busy queues and FreePBX, it often results in requests for help in the forums and to our support department because of overloaded systems or certain functions that do not seem work right. FreePBX was designed to be an SMB PBX concentrating on feature rich capabilities at the expense of complex dialplan and AGI scripts, all of which take their toll when you setup a reasonable size call center and start throwing lots of simultaneous calls at a group of queue members. This Technical Corner post will discuss some of the advancements introduced in FreePBX over the years to help accommodate systems that make extensive use of queues. These same changes will have positive impact on low power appliances and any other system that uses queues.

Throughout the article we will refer to FreePBX Queue Agents as members to be consistent with the Asterisk naming convention. FreePBX commonly refers to any queue member as a Queue Agent which is someone who answers calls destined for the queue and does not specifically refer to the Asterisk agents.conf agents.

Before going into the solutions we will frame the problem and describe why FreePBX runs into issues that other Asterisk configurations may not. The root cause of the issues described in this article hinge on two fundamentals that are employed by FreePBX. First of all, FreePBX abstracts extensions from the device(s) they are associated with. You can read more about this in a past article FreePBX Devices and Users Under the Hood. As a result, queue members are not dialed by calling the device directly but instead are sent down the Asterisk dialplan through the Local channel. This allows the extension to be called in the same way that they would be dialed by another user on the system. It also enables all possible features such as call forwarding, follow-me, etc. It further allows arbitrary external phone numbers to be treated as members in addition to other creative uses. This leads to the second fundamental, all internal calls in FreePBX employ the use of the dialparties.agi AGI script along with other AGI scripts when making calls. As a result, calling a queue member is a “heavy” operation (as is calling any extension on the system). In the case of a queue, this is multiplied when employing a ringall strategy (very common) because every queue call that is dispatched ends up in simultaneous calls to every member, and thus a barrage of heavy dialplan and AGI scripts are launched, often just to find out that most of the members are currently in use and should not have their phones rung anyhow.

Aside from overloading the system when many members are present, the most common side effect of using the Local channel when making calls is encountered when a member transfers their received call elsewhere. When this occurs in conjunction with the Asterisk queues configuration of ringinuse=no (see below), that member remains “Busy” until that transferred call is terminated. This occurs because of the nature of the Local channel. This means that the member is now free but will not receive any calls from the queue.

So what has FreePBX done to address both these common problems, as well as get around the fact that many of the desirable features of the PBX are problematic when an extension is included as part of a queue? To answer the latter question, there have been various features added to the queue module, often features that are outside of those provided by the underlying Asterisk queue application. The most important additions have included:

  • Agent Restrictions
    • Call as Dialed
      Send all calls down the standard FreePBX Local channel
    • No Follow-Me or Call Forward
      Restrict calls to known extensions down a specific context which blocks all Call Forward and avoids Follow-Me configurations
    • Extensions Only
      Allow only extensions to be dialed, and send them down a special context that puts further restrictions on them. Certain safe guards are removed from the queue calls that normally help block calls from hitting member voicemails since the special context handles that. As a result, this mode is also safe for direct device dialing.
  • Skip Busy Agent
    • No
      Occupied members will have new queue calls presented to them.
    • Yes
      Occupied members will be treated as if they did not have call waiting. The calls will be dispatched to them (resulting in system load) but the calls will be blocked within dialparties.agi (or macro-dial-one if using this new capability described below.)
    • Yes + (ringinuse=no)
      Same situation as above with the addition of the queues.conf setting of ringinuse=no. With no patches and standard Local channel members, this will result in calls not being dispatched to busy members from this queue including those who subsequently transferred their calls elsewhere (an undesirable side effect). With the patch below installed, this side effect goes away.
    • Queue Calls Only (ringinuse=no)
      Without the patch below, this will result in the queue not sending calls to members currently on calls from this or another queue. It has the same side effects as the previous case. If there are no queue calls the call will be dispatched and if the member is on their phone with a non-queue call, their phone will still ring with this call (unless they have call waiting disabled). When the patch is introduced, this setting is functionally equivalent to the previous one.

These settings provide various levels of filtering to avoid undesirable features such as different Call Forward and Follow-Me features, or multi-line phones ringing additional queue calls when members are busy on existing calls.

Addressing the two main issues of this article, overloaded systems and members seen as busy after a call transfer takes a bit more work and often some patches to Asterisk that we have developed and submitted to the Asterisk bug tracker and available for anyone to take advantage of. We will first discuss the member who remains busy after a call transfer as the solution to that problem is connected to the system overload issue. The crux of the problem is that the queue members are viewed as Local channels, in asterisk terms, they are represented in the queues_additional.conf file as:

member=Local/2201@from-internal/n,0

or something similar. So once the queue sends a call down to that member, even if the call is subsequently transfered, the original Local channel is still busy and the queue application does not have any more information to determine the state of that member, so they remain busy and no additional calls are sent to that member if the Asterisk queue configuration ringinuse=no is set. The Asterisk team recognized the limitation in version 1.6 and introduced some additional parameters to the member assignment. They introduced the ability to associate a different device state with the queue member. They also recognized the limitation as enough of an issue that they made a rare move and added the new “feature” as a back port starting in Asterisk 1.4.25. This was a great move however the solution was incomplete and does not become complete until Asterisk 1.8 (unless enough of you complain). Adding the ability to use the state of a device is nice, but it misses the fact that the state of an extension is not solely based on a device state. It is based on the state of the “hint” associated with that extension and could include multiple devices as well as custom extension state information (such as a server side DND feature). Since all extensions in FreePBX are abstracted and use such hints to determine their state, this new ability can not be employed without patching asterisk to make their solution complete. (A hint in Asterisk is what is used to provide status to telephone BLF buttons and implement more sophisticated dialplan, see Asterisk documentation for more information on the hint priority).

Not wanting to miss out on the this new capability which comes so close to solving the problem, we set out to “finish” what Asterisk started and submitted a patch against Asterisk 1.4 in May of 2009 that would add the “device” state” of HINT:nnn@context as a way to associate a hint vs. a real device. If this patch is applied to your Asterisk install, and the secret “USEQUEUESTATE=yes” amportal.conf variable is set, FreePBX will generate the dialplan and configuration files to associate the extension hints with the queue members. As a result, the “member transferred call” previously discussed will now result in the member appearing available by the queue again, since their hint will no longer show busy. (In addition, it ties into other enhancements. For example, if your system employs server side DND, and you have included the 1.4 back port of DEVICE_STATE() that is common on many 1. 4 system, and set the secret FreePBX amportal.conf variable of “USEDEVSTATE=true” then your member will be seen as busy by the queue and calls will not be attempted to the them. Note, the back port of this patch has not been modified for Asterisk 1.6 and as of this writing, the Asterisk team has chosen to delay this capability until Asterisk 1.8. We feel this is a big mistake as the original “fix” introduced by Asterisk was an incomplete solution in that it completely ignored the fundamental state mechanism that Asterisk is designed around, the hint priority.)

Now back to the overloaded queue situation discussed earlier. The crux of the issue occurs when you don’t set Skip Busy Agent to Yes + (ringinuse=no) or Queue Calls Only (ringinuse=no). Doing options other than these two tells the queue to try all members regardless of whether they are on the phone or not. Without the above patch, this gets around the “member transferred call” issue but results in all members being called whether or not they are busy. The underlying dialplan still blocks the member’s phone from ringing in dialparties.agi, however the system can be loaded heavily because of the barrage of dialplan and AGI scripts being run. When the above patch is applied, you now remove that barrier resulting in a solution that addresses both issues. By setting either of these two settings in conjunction with the member hint information supplied by the patch, the queue will not overload the system with attempted calls to any member seen as in use based on their hint information. This can take a tremendous load off the system while maintaining proper state information and avoiding in use members from receiving calls that will just be rejected by dialparties.agi. Now if the Asterisk team would recognize the limitation of their original fix and back port the “HINT:” solution to 1.4 and 1.6 we could all benefit from this ability without the need to patch Asterisk!

This new Asterisk capability in conjunction with the required patch solves the “transfer” problem and reduces the number of members being called to those who are actually available to take a call. That is often enough to solve the problem on heavily loaded systems since in most call center environments members who are not available will either have their phones set to DND or use the Dynamic Member ability of queues and logout when they should not be taking calls. However, for those members who are busy, a call is still sent down the Local channel for each member which results in heavy dialplan and the dialparties.agi script being run for each member. This may still be a problem in some environments so more extreme actions may be necessary.

If the above solutions are not adequate for some installations, or patching Asterisk is not an option, there are two new capabilities that have were introduced in FreePBX 2.7. These are designed for advanced users who are ready to thoroughly test their installations to assure that other system functions are not compromised. The first one is the ability to use device in place of extensions for queue members. The ability to represent an extension as “Annn” has existed for a few years and results in the the queue agent referencing channel Agent/nnn. This has always been an experimental ability with known limitations and no GUI support to configure the required agents.conf configuration file. Asterisk has deprecated the channel Agent in 1.4 and no longer supports it in 1.6. FreePBX 2.7 has introduced syntax that allows SIP, IAX2, DAHDI and ZAP devices to be referenced directly. This is achieved by prefacing the member number with S, X, D or Z respectively. This capability is only available for static members. When this is done the queue calls are dispatched direct to the specified device completely bypassing the FreePBX dialplan. In the past, doing this resulted in broken features such as subsequent transfers to voicemail hanging up on the caller. By setting “Agent Restrictions” to “Extensions Only” this problem is now eliminated. This results in all of the dialplan including the heavy use of dialparties.agi being avoided. It should be understood that avoiding the dialplan also means that features will also be avoided, such as an extension that is set to always record calls will not have the calls recorded unless other settings ultimately enable this.

The device specific members are limited to Static Agents and may not be desirable for this or other reasons. We have thus introduced an experimental macro that is ultimately targeted for general release in FreePBX 2.8 but accessible in 2.7 with some magic settings. This macro, macro-dial-one, serves as a replacement for macro-dial + dialparties.agi in all cases where a single extensions is being called. This is always the case when the queue is dispatching calls. We have included the macro in 2.7 in order to get early testing since we plan it to replace macro-dial + dialparties.agi for single extension dialing in 2.8. Enabling the macro has two requirements. First, you must set the magic variable USEDIALONE=true in amportal.conf. Secondly, if running Asterisk 1.4 you must have the 1.4 back port of the function EXTENSION_STATE() that is not officially available until 1.6. The advantage of this macro is the elimination of the dialparties.agi script which results in a forked process and relatively heavy php environments being run for each member that is being called. The call is still sent down the Local channel and results in a fair amount of Asterisk dialplan executing but avoids the heavy system load created by expensive process forking and php AGI scripts. More details and a copy of the func_extstate.c can be obtained in the macro-dial-one trac ticket. It should be stressed that the macro is at an alpha level. We hope people will test it and report issues or patches to address bugs as they are discovered to help prepare this for the 2.8 release. Setting USEDIALONE=true results in all single extension dialing to use this macro, not just the queue.

There have been many other enhancements included with queues and not listed here to improve the overall capabilities and flexibilities of the Queue Module as well as other articles written in this Technical Corner series about queues. For now, we hope the information presented is useful for those of you who have busy queues and are looking for ways to optimize your environment as well as those of you who are just interested in knowing more about the inner workings of FreePBX + Asterisk and what makes it all tick!

Philippe – On behalf of the FreePBX Team

11 thoughts on “Heavy Queue Usage in FreePBX

  1. Yes Philippe, thanks for all you have done. If this works it will make FreePBX based Asterisk even more viable as a solution for small call centers. We will be testing this in the near future.
  2. I manuelly change the queues_additional.conf and add member=Local/50001@from-queue/n,0,QueueMember,hint:50001@ext-local to queue 8001,aslo copy the from-queue extension from freepbx 2.8,i have two questions.
    1.when i call 8001 and the 50001 extension ringing,does it mean all things goes right?Can i avoid the 2 issues metioned above?Does usequeuestate=yes is only the settings for freebpx to generate the conf file but the asterisk donn’t support it?

    2.If i change to member=Local/50001@from-interanl/n,0,QueueMember,hint:50001@ext-local,50001 also ringing,what’s the difference between from-internal and from-queue?Does it is a solution for the system who doesn’t do the patches?

    Code;
    [from-queue]
    include => from-queue-custom
    exten => 8001,1,Goto(from-internal,${QAGENT},1)
    exten => _.,1,Set(QAGENT=${EXTEN})
    exten => _.,n,Goto(${NODEST},1)

    ; end of [from-queue]

  3. Asterisk trunk and 1.4 branch are going to be substantially different as trunk is basically 1.8+

    you should be able to check the Asterisk ticket with the original patch as there may have been some updated contributions to it from minor changes.

  4. I have applyed the app_queue.c patch to AS 1.4.38 and use “USEQUEUESTATE=yes” in amportal.conf. How to test if it works?Does it harmful if i doesn’t change the php files?

    And what’s the patch to trixbox’s file?Could you show me what does freebpx changed?I will apply it to trixbox.And if i do it successfully,i will write a guide about it.

    which rev do the change in http://www.freepbx.org/trac/browser/freepbx

  5. this is an Asterisk patch, it has nothing to do with trixbox beyond Asterisk changes and if you patched Asterisk, then you have already ‘broken’ aspects of tirxbox since you require source to apply the patch.

    You can check if the correct dialplan is being generated by checking the queues_additional.conf and see if your members settings are using the “HINT:…” state information.

  6. i use the newest asterisk 1.4.38 core,and the source of app_queue.c has a lot of changes to the one in /[asterisk]/trunk/apps/app_queue.c which contains the patch of yours.can i simply use the trunk of instead or should i add the patch code below?

    — trunk/apps/app_queue.c 2009/10/06 01:24:24 222176
    +++ trunk/apps/app_queue.c 2009/11/03 21:16:14 227424
    @@ -824,20 +824,22 @@
    };

    struct member {
    – char interface[80]; /*!< Technology/Location to dial to reach this member*/ - char state_interface[80]; /*!< Technology/Location from which to read devicestate changes */ - char membername[80]; /*!< Member name to use in queue logs */ - int penalty; /*!< Are we a last resort? */ - int calls; /*!< Number of calls serviced by this member */ - int dynamic; /*!< Are we dynamically added? */ - int realtime; /*!< Is this member realtime? */ - int status; /*!< Status of queue member */ - int paused; /*!< Are we paused (not accepting calls)? */ - time_t lastcall; /*!< When last successful call was hungup */ - struct call_queue *lastqueue; /*!< Last queue we received a call */ - unsigned int dead:1; /*!< Used to detect members deleted in realtime */ - unsigned int delme:1; /*!< Flag to delete entry on reload */ - char rt_uniqueid[80]; /*!< Unique id of realtime member entry */ + char interface[80]; /*!< Technology/Location to dial to reach this member*/ + char state_exten[AST_MAX_EXTENSION]; /*!< Extension to get state from (if using hint) */ + char state_context[AST_MAX_CONTEXT]; /*!< Context to use when getting state (if using hint) */ + char state_interface[80]; /*!< Technology/Location from which to read devicestate changes */ + char membername[80]; /*!< Member name to use in queue logs */ + int penalty; /*!< Are we a last resort? */ + int calls; /*!< Number of calls serviced by this member */ + int dynamic; /*!< Are we dynamically added? */ + int realtime; /*!< Is this member realtime? */ + int status; /*!< Status of queue member */ + int paused; /*!< Are we paused (not accepting calls)? */ + time_t lastcall; /*!< When last successful call was hungup */ + struct call_queue *lastqueue; /*!< Last queue we received a call */ + unsigned int dead:1; /*!< Used to detect members deleted in realtime */ + unsigned int delme:1; /*!< Flag to delete entry on reload */ + char rt_uniqueid[80]; /*!< Unique id of realtime member entry */ }; enum empty_conditions { @@ -1258,6 +1260,81 @@ } } +/*! \brief Helper function which converts from extension state to device state values */ +static int extensionstate2devicestate(int state) +{ + switch (state) { + case AST_EXTENSION_NOT_INUSE: + state = AST_DEVICE_NOT_INUSE; + break; + case AST_EXTENSION_INUSE: + state = AST_DEVICE_INUSE; + break; + case AST_EXTENSION_BUSY: + state = AST_DEVICE_BUSY; + break; + case AST_EXTENSION_RINGING: + state = AST_DEVICE_RINGING; + break; + case AST_EXTENSION_ONHOLD: + state = AST_DEVICE_ONHOLD; + break; + case AST_EXTENSION_UNAVAILABLE: + state = AST_DEVICE_UNAVAILABLE; + break; + case AST_EXTENSION_REMOVED: + case AST_EXTENSION_DEACTIVATED: + default: + state = AST_DEVICE_INVALID; + break; + } + + return state; +} + +static int extension_state_cb(char *context, char *exten, enum ast_extension_states state, void *data) +{ + struct ao2_iterator miter, qiter; + struct member *m; + struct call_queue *q; + int found = 0, device_state = extensionstate2devicestate(state); + + qiter = ao2_iterator_init(queues, 0); + while ((q = ao2_iterator_next(&qiter))) { + ao2_lock(q); + + miter = ao2_iterator_init(q->members, 0);
    + for (; (m = ao2_iterator_next(&miter)); ao2_ref(m, -1)) {
    + if (!strcmp(m->state_context, context) && !strcmp(m->state_exten, exten)) {
    + update_status(q, m, device_state);
    + ao2_ref(m, -1);
    + found = 1;
    + break;
    + }
    + }
    + ao2_iterator_destroy(&miter);
    +
    + ao2_unlock(q);
    + ao2_ref(q, -1);
    + }
    + ao2_iterator_destroy(&qiter);
    +
    + if (found) {
    + ast_debug(1, “Extension ‘%s@%s’ changed to state ‘%d’ (%s)\n”, exten, context, device_state, ast_devstate2str(device_state));
    + } else {
    + ast_debug(3, “Extension ‘%s@%s’ changed to state ‘%d’ (%s) but we don’t care because they’re not a member of any queue.\n”,
    + exten, context, device_state, ast_devstate2str(device_state));
    + }
    +
    + return 0;
    +}
    +
    +/*! \brief Return the current state of a member */
    +static int get_queue_member_status(struct member *cur)
    +{
    + return ast_strlen_zero(cur->state_exten) ? ast_device_state(cur->state_interface) : extensionstate2devicestate(ast_extension_state(NULL, cur->state_context, cur->state_exten));
    +}
    +
    /*! \brief allocate space for new queue member and set fields based on parameters passed */
    static struct member *create_queue_member(const char *interface, const char *membername, int penalty, int paused, const char *state_interface)
    {
    @@ -1277,7 +1354,14 @@
    ast_copy_string(cur->membername, interface, sizeof(cur->membername));
    if (!strchr(cur->interface, ‘/’))
    ast_log(LOG_WARNING, “No location at interface ‘%s’\n”, interface);
    – cur->status = ast_device_state(cur->state_interface);
    + if (!strncmp(state_interface, “hint:”, 5)) {
    + char *tmp = ast_strdupa(state_interface), *context = tmp;
    + char *exten = strsep(&context, “@”) + 5;
    +
    + ast_copy_string(cur->state_exten, exten, sizeof(cur->state_exten));
    + ast_copy_string(cur->state_context, S_OR(context, “default”), sizeof(cur->state_context));
    + }
    + cur->status = get_queue_member_status(cur);
    }

    return cur;
    @@ -2677,7 +2761,7 @@
    tmp->stillgoing = 0;

    ao2_lock(qe->parent);
    – update_status(qe->parent, tmp->member, ast_device_state(tmp->member->state_interface));
    + update_status(qe->parent, tmp->member, get_queue_member_status(tmp->member));
    qe->parent->rrpos++;
    qe->linpos++;
    ao2_unlock(qe->parent);
    @@ -2760,7 +2844,7 @@
    ast_channel_unlock(qe->chan);
    do_hang(tmp);
    (*busies)++;
    – update_status(qe->parent, tmp->member, ast_device_state(tmp->member->state_interface));
    + update_status(qe->parent, tmp->member, get_queue_member_status(tmp->member));
    return 0;
    } else if (qe->parent->eventwhencalled) {
    char vars[2048];
    @@ -2788,7 +2872,7 @@
    ast_channel_unlock(tmp->chan);
    ast_channel_unlock(qe->chan);

    – update_status(qe->parent, tmp->member, ast_device_state(tmp->member->state_interface));
    + update_status(qe->parent, tmp->member, get_queue_member_status(tmp->member));
    return 1;
    }

    @@ -6093,7 +6177,7 @@
    */
    q->membercount++;
    }
    – member->status = ast_device_state(member->state_interface);
    + member->status = get_queue_member_status(member);
    return 0;
    } else {
    q->membercount–;
    @@ -7480,6 +7564,8 @@
    if (device_state_sub)
    ast_event_unsubscribe(device_state_sub);

    + ast_extension_state_del(0, extension_state_cb);
    +
    if ((con = ast_context_find(“app_queue_gosub_virtual_context”))) {
    ast_context_remove_extension2(con, “s”, 1, NULL, 0);
    ast_context_destroy(con, “app_queue”); /* leave no trace */
    @@ -7553,6 +7639,8 @@
    res = -1;
    }

    + ast_extension_state_add(NULL, NULL, extension_state_cb, NULL);
    +
    ast_realtime_require_field(“queue_members”, “paused”, RQ_INTEGER1, 1, “uniqueid”, RQ_UINTEGER2, 5, SENTINEL);

    return res ? AST_MODULE_LOAD_DECLINE : 0;

  7. 1.4.24 is fine.

    There’s a fair amount of detail in the post.

    If you need more detail, you may want to consider engaging the paid support team (or someone else equivalent) to further help you as queues can get pretty complex if you are running into issues that need the type of help listed here.

Leave a Reply