Beta testers wanted for new CDR report tool (UPDATED)

mickecarlsson's picture

UPDATE: 2011-08-22, beta2 version released.
UPDATE: 2011-08-11, beta version released.

I have found a new nice report program for Asterisk CDR. It is written by Anthony Joseph Messina, modified by Igor Okunev and it is called Asterisk CDR Viewer.

I rewrote the code into a FreePBX Module. The code is in alpha stage now, there might be some quirks in it that I have not thought about. My tests has been on a rather large CDR database that contains records dated from 2006 up to today with no strange effects.

Download the BETA FreePBX CDR Viewer Upload it via Module Admin.
You will find it in Tools section with the title CDR Viewer.
It will work with FreePBX 2.7 and up.
Note that ticking Concurrent calls and run a report will take some (considerable) time.

Give it some testing and let me know what you think.
What I like to know is Can this module replace the current report tool that FreePBX use today?

In this release only .wav files are supported, that might change in later releases.


__________________

Mikael Carlsson
(I am off-line, tinkering with my Chevy and my radios, don't know when I will be back)


Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

It looks pretty neat. Did

cosmicwombat's picture

It looks pretty neat. Did not appear to break anything on a FreePBX ISO install. I am not sure I am seeing a way to search on DID dialed as to track inbound calls. Then again that might require some other manipulation.


Can you add grouping of

black187's picture

Can you add grouping of numbers in it (Let's say two companies have one FreePBX server, but the CDR has to be made for two seperate group of numbers)?


working great

mlungu's picture

Nice addition - I see on the homepage that you can add links to the recordings. The instructions on their website relates to an installation done outside of freepbx. How would I edit (or where would I find it rather) the config.inc.php file that relates to this when installed as a freepbx module?

And has anybody tried linking the sound files already?

Thanks for any input


cosmicwombat: DID will (most

mickecarlsson's picture

cosmicwombat: DID will (most likely) be added to 2.10, but that requires a configuration change switching from cdr_mysql for CDR to res_odbc/cdr_adaptive_odbc.conf. By doing that we can add whatever field we want to CDR. This module will be updated when more testing have been done.

black187: Can you elaborate the question?

mlungu: All settings in config.inc.php are now read from FreePBX using the settings that already exists. However, a bug prevents that (my bad).


__________________

Mikael Carlsson
(I am off-line, tinkering with my Chevy and my radios, don't know when I will be back)


Done - but how do I see the call recordings?

mlungu's picture

Hi Mikael,

Uploaded the new one - do you have any idea how I would now see the link to the call recording??


Today I finally had the time

mickecarlsson's picture

Today I finally had the time to really test on a live environment, there are a few more bugs to sort out. And that is why the module is still in alpha stage:-)

Give me a couple of days to test the link to recordings function and add code so that one can play or download it.


__________________

Mikael Carlsson
(I am off-line, tinkering with my Chevy and my radios, don't know when I will be back)


ok ok ok

mlungu's picture

No problem :-) I know it's still alpha. I just get excited a bit easily :-). Your modules are highly appreciated!


Regarding grouping of number

black187's picture

@mickecarlsson: The idea is - when you click on the CDR viewer, you can choose or make a group, which contains some extensions. And you can then view the CDR records only for the selected/made group.


@black187, oh, I see. Hmmm,

mickecarlsson's picture

@black187, oh, I see. Hmmm, let me think about that. No promises made :-)


__________________

Mikael Carlsson
(I am off-line, tinkering with my Chevy and my radios, don't know when I will be back)


multiple asterisk-regexp for src/dst fields

igor_okunev's picture

I made small changes to support multiple regular expressions separated by a comma in the fields SRC/DST.
For example:
* src = '_2XXN, _562., _.0075' - src any of these numbers
* src = '_!2XXN, _562., _.0075' - src any other than these numbers

If it is normal for you, you can add this changes to the module.

Index: include/config.inc.php
===================================================================
--- include/config.inc.php (revision 50)
+++ include/config.inc.php (working copy)
@@ -170,7 +170,7 @@
$number = substr($number,1);
} elseif ( '_' == substr($number,0,1) ) {
$number_chars = preg_split('//', substr($number,1), -1, PREG_SPLIT_NO_EMPTY);
- $number = '^';
+ $number = '';
foreach ($number_chars as $chr) {
if ( $chr == 'X' ) {
$number .= '[0-9]';
@@ -187,7 +187,6 @@
}
}
$_POST[ $source_data .'_mod' ] = 'asterisk-regexp';
- $number .= '$';
}
return $number;
}
Index: index.php
===================================================================
--- index.php (revision 50)
+++ index.php (working copy)
@@ -116,7 +116,22 @@
}
break;
case "asterisk-regexp":
- $$key = "AND $key $pre_like RLIKE '$val[0]'";
+ $ast_dids = preg_split('/\s*,\s*/', $val[0], -1, PREG_SPLIT_NO_EMPTY);
+ $ast_key = '';
+ foreach ($ast_dids as $did) {
+ if (strlen($ast_key) > 0 ) {
+ if ( $pre_like == ' NOT ' ) {
+ $ast_key .= " and ";
+ } else {
+ $ast_key .= " or ";
+ }
+ if ( '_' == substr($did,0,1) ) {
+ $did = substr($did,1);
+ }
+ }
+ $ast_key .= " $key $pre_like RLIKE '^$did\$'";
+ }
+ $$key = "AND ( $ast_key )";
break;
case "begins_with":
default:


Community, please welcome

mickecarlsson's picture

Community, please welcome Igor Okunev, the author behind Asterisk CDR Viewer.

@igor_okunev, I have updated FreePBX svn with the above changes.

For those who wants to test out the new feature, download page.cdr.php and functions.inc.php and replace those in modules/cdr.
Run amportal chown after the files have been replace to set correct owner for the files.


__________________

Mikael Carlsson
(I am off-line, tinkering with my Chevy and my radios, don't know when I will be back)


The CDR Viewer is now

mickecarlsson's picture

The CDR Viewer is now released as beta.
Download the BETA FreePBX CDR Viewer Upload it via Module Admin.
You will find it in Tools section with the title CDR Viewer.

In this release you can play recorded calls. NOTE: Only .wav is supported at the moment.


__________________

Mikael Carlsson
(I am off-line, tinkering with my Chevy and my radios, don't know when I will be back)


New version seemed to

michigantelephone's picture

New version seemed to install fine over earlier alpha5 version on FreePBX 2.9 system (FreePBX distro) but on a FreePBX 2.8/Elastix system, when the orange box came up the frog never appeared, nor did any text at all - it just froze at that point (I left it alone for several minutes) and it stayed that way until I closed that page and opened another. I then completely uninstalled the alpha5 version, and tried to install the new version with the same result. All indications are that the install never actually begins.

I re-uploaded alpha5 on that system and it installed without a hitch. Is the new version perhaps limited to FreePBX 2.9 and above?


No, but there are code in it

mickecarlsson's picture

No, but there are code in it that is PHP 5 only. Hmm, I need to check that before installation.
Could you check the php used in that system that failed? Do it with php -v in a shell?

Thanks for the report, I will try to install it on an Elastix system and see what happens.

Update: Now that I think of it, it must be the update.php scrip that fails, it tries to add a field in the asteriskcdrdb/cdr database/table.
I definitely needs to test that.


__________________

Mikael Carlsson
(I am off-line, tinkering with my Chevy and my radios, don't know when I will be back)


# php -v PHP 5.2.17 (cli)

michigantelephone's picture

# php -v
PHP 5.2.17 (cli) (built: Aug 9 2011 20:12:01)
Copyright (c) 1997-2010 The PHP Group
Zend Engine v2.2.0, Copyright (c) 1998-2010 Zend Technologies

Obviously that's been updated since the original install (yesterday, in fact, because of the security issue).

I just checked and it did not succeed in adding the field. I wonder if there's a user/password issue, perhaps?


OK, that pointed me to the

mickecarlsson's picture

OK, that pointed me to the solution, could you edit install.php from this:

// Update cdr database with did field
//
$sql = "SELECT did FROM asteriskcdrdb.cdr";
$confs = $db->getRow($sql, DB_FETCHMODE_ASSOC);

to this:

// Update cdr database with did field
//
global $db;
$sql = "SELECT did FROM asteriskcdrdb.cdr";
$confs = $db->getRow($sql, DB_FETCHMODE_ASSOC);

I forgot that we always include that from 2.9 and onwards.


__________________

Mikael Carlsson
(I am off-line, tinkering with my Chevy and my radios, don't know when I will be back)


This time the orange box

michigantelephone's picture

This time the orange box came up and it said "Please wait while module actions are performed" and that was it… no frog or other text. It just sits there. Again I totally uninstalled the previous version (even deleted the cdr directory after doing that), uploaded the new version, and made the change you requested.

I decided to try an experiment and added a few debug lines:

// Update cdr database with did field
//
out("debug 1");
global $db;
out("debug 2");
$sql = "SELECT did FROM asteriskcdrdb.cdr";
out("debug 3");
$confs = $db->getRow($sql, DB_FETCHMODE_ASSOC);
out("debug 4");
if (DB::IsError($confs)) { // no error... Already there

NOTHING appeared in the orange box. So for some reason, it sure looks like the install script isn't being executed at all. Yet as I say, the older alpha version installs fine (but there is no install.php in that one, it appears)


OK, I will look at it later

mickecarlsson's picture

OK, I will look at it later tonight and install it on an old 2.8 setup. Thanks for the feedback.


__________________

Mikael Carlsson
(I am off-line, tinkering with my Chevy and my radios, don't know when I will be back)


You're welcome. I found the

michigantelephone's picture

You're welcome. I found the answer, see my previous post (which will probably appear BELOW this one because I forgot to attach it as a reply, and apparently there's no way to delete a message if you did it wrong).


GOT IT! You were on the

michigantelephone's picture

GOT IT!

You were on the right track, you just needed to swipe a little more code from a similar install.php. THIS worked:

// Update cdr database with did field
//
global $db;
global $amp_conf;
if (! function_exists("out")) {
        function out($text) {
                echo $text."<br />";
        }
}
$sql = "SELECT did FROM asteriskcdrdb.cdr";
$confs = $db->getRow($sql, DB_FETCHMODE_ASSOC);

Note that JUST adding "global $amp_conf;" did NOT do the trick, so the "out" function thing must have been the key.


I spotted that but ran out

mickecarlsson's picture

I spotted that but ran out of time before heading out to work. The $amp_conf is not needed as we don't have any calls to retrieve amp_conf variables.
I have republished the module with all fixes, please download it again and test.
Thanks for the feedback.
Have you tested the playback of recordings?


__________________

Mikael Carlsson
(I am off-line, tinkering with my Chevy and my radios, don't know when I will be back)


The revised one installs

michigantelephone's picture

The revised one installs fine.

Haven't tested playback of recordings because I've never actually had a reason to create a recording (I assume you mean a recoding of a call, and I don't even remember the code for that offhand), therefore there are none on the system, and I need to leave in a few minutes. Maybe later tonight I will give that a try.


Thanks for the feedback. As

mickecarlsson's picture

Thanks for the feedback. As for recording, just set an extension to Record Incoming to Always and dial into it a couple of times. Then set it back to what it was.

Any other takers for testing the playback of recorded calls?


__________________

Mikael Carlsson
(I am off-line, tinkering with my Chevy and my radios, don't know when I will be back)


Okay, I tested playback of

michigantelephone's picture

Okay, I tested playback of recordings on a Mac (OS X 10.5.8) using Firefox, Safari, Chrome, and Opera. In Firefox, Safari and Chrome they played fine using the QuickTime plug-in. In Opera they played fine also although some of the controls were not available, for example you couldn't pause the recording. I think this may be an Opera or a QuickTime issue.

Also tested on Firefox, Chrome and Opera on Ubuntu Linux 10.10. In Firefox it plays fine there also, using the "Totem Browser Plug-In". In Chrome and Opera, it just says "Missing Plug-in" where the control should be. There's probably a fix for that, but since I don't use that system for web browsing I'm not going to worry about it, and I'm inclined to think it's not a problem with your software because a quick Google search tells me that a LOT of people have problems getting sound to play in Ubuntu, when using browsers other than Firefox (and Google Chrome users apparently even have issues getting sound to play under some versions of Windows, so there's that).


Just a note for anyone

michigantelephone's picture

Just a note for anyone testing this that wants to see the DID field actually populated. For now you can add a context at the bottom of /etc/asterisk/extensions_custom.conf

[custom-from-trunk-did]
exten => _X!,1,Set(CDR(did)=${EXTEN})
exten => _X!,n,Goto(from-trunk,${EXTEN},1)
exten => h,1,Macro(hangupcall,)

Then in any trunk that handles incoming calls, set the context statement to context=custom-from-trunk-did (assuming the statement was originally context=from-trunk).

If the context statement in a trunk points to something other than from-trunk, there are a couple of ways to handle it. If it went to another custom context that you created, just add a line to that custom context that includes the Set(CDR(did)=${EXTEN}) instruction. If you went to some other valid internal context (for example, from-pstn-e164-us, from-pstn-toheader, etc.) just create another corresponding context in extensions_custom conf, for example:

[custom-from-pstn-e164-us-did]
exten => _X!,1,Set(CDR(did)=${EXTEN})
exten => _X!,n,Goto(from-pstn-e164-us,${EXTEN},1)
exten => h,1,Macro(hangupcall,)

And then change the trunk context line to point to the appropriate custom context. But keep in mind that whenever this module becomes part of the official distribution, code to set the DID field in the CDR records will probably be added to from-trunk and similar contexts, making these custom contexts redundant at best.

Testing on PIAF

Snisk's picture

Ran into an issue testing the CDR Viewer on my PBX in a Flash setup FreePBX 2.9. If I raise the result limit to 10,000 or higher and check the export to CSV it crashes (after about 10 seconds). However if I try to do the 10,000 and only check the cdr search it works.


Well, I can put in a limit

mickecarlsson's picture

Well, I can put in a limit on that, 10,000 entries are a lot of entries:-)
I guess that memory is exhausted in php and that is why it crashes. Is there any error in apache error_log?

Update:
PHP Fatal error: Maximum execution time of 30 seconds exceeded in /var/www/html/admin/modules/cdr/crypt.php on line 0, referer: http://192.168.0.205/admin/config.php?type=tool&display=cdr

Could you test this, in functions.inc.php add the following from:

function cdr_export_csv($csvdata) {
        global $db;
        $fname          = "cdr__" .  (string) time() . $_SERVER["SERVER_NAME"] . ".csv";

to

function cdr_export_csv($csvdata) {
        global $db;
        set_time_limit(3000);
        $fname          = "cdr__" .  (string) time() . $_SERVER["SERVER_NAME"] . ".csv";

Then test it again.

Update 2:
Just tested it with 10,000 records, it took almost a minute, then I could download it.

Thanks for the report.


__________________

Mikael Carlsson
(I am off-line, tinkering with my Chevy and my radios, don't know when I will be back)


Thanks It works now without

Snisk's picture

Thanks It works now without crashing. However took about 2 min on the system I am running. Bad thing is that I need to create a csv of our monthly call logs for reporting. So I went directly to the database and queried the past month August 1st through August 31 I have 62,000 + lines!!!! So any suggestions on a better way. I was testing making a cron job to query the DB and then upload it to my FTP but kept getting issues with too many arguments. Its been a long week so maybe I am missing something.. Thanks for the help in advance


Unfortunately, it will take

mickecarlsson's picture

Unfortunately, it will take some time to process 62,000+ lines.
I wonder how long it took for the old cdr report tool to process that. Hmmm, I need to test it.


__________________

Mikael Carlsson
(I am off-line, tinkering with my Chevy and my radios, don't know when I will be back)


The old reports take a long

w5waf's picture

The old reports take a long time too. Taking 2+ minutes would not be unreasonable.

BF


__________________

Bill/W5WAF
IT Specialist
City of Vicksburg, MS


I have a CDR DB with 7M+

SkykingOH's picture

I have a CDR DB with 7M+ records. Would you like me to see what happens with that data set?


That was 2+ minutes on the

Snisk's picture

That was 2+ minutes on the 10,000. The issue is that some months we may have 60,000 lines and others we may have 70,000 or even 80,000 lines. When I query the Database manually and email to my self it creates a ~13Meg file of all 60,000+ lines, in about 1 min and then I have it in my Personal Inbox (Not PBX Mailbox) within another minute or 2.


I am looking into it, it is

mickecarlsson's picture

I am looking into it, it is the foreach loop that is taking time.
If I select 10,000 records, the select for that take about 0,4 seconds, the foreach loop that populates the csv take 43 seconds.
I will try to see if I can optimize the code foreach or do another approach.

@SkykingOH, I have a rather large cdr database to play with:

mysql> select count(*) from cdr;
+----------+
| count(*) |
+----------+
|   879652 |
+----------+

__________________

Mikael Carlsson
(I am off-line, tinkering with my Chevy and my radios, don't know when I will be back)


OK, I have now tweaked the

mickecarlsson's picture

OK, I have now tweaked the code a bit, I found where the time was spent in the code.

With 10,000 records, it took about 50 seconds to generate the csv export, with this code it will take about 0,6 seconds to export to csv.

In functions.inc.php replace the function cdr_export_csv with this code:

function cdr_export_csv($csvdata) {
        set_time_limit(3000);
        $fname          = "cdr__" .  (string) time() . $_SERVER["SERVER_NAME"] . ".csv";
        $csv_header ="calldate,clid,src,dst,dcontext,channel,dstchannel,lastapp,lastdata,duration,billsec,disposition,amaflags,accountcode,uniqueid,userfield\n";
        $data           = $csv_header;

        foreach ($csvdata as $csv => $value)
        {
                $tmp = '';
                foreach ($value as $csv_key => $csv_val)
                {
                    // If the string contains a comma, enclose it in double-quotes.
                    if (strpos($csv_val, ",") !== FALSE) {
                        $csv_val = "\"" . $csv_val . "\"";
                    }
                        $tmp .= ",$csv_val";
                }
                $data .= substr($tmp, 1)."\n";
        }

        cdr_download($data, $fname);
        return;
}

Please test and report back.


__________________

Mikael Carlsson
(I am off-line, tinkering with my Chevy and my radios, don't know when I will be back)


On my system it seems to

Snisk's picture

On my system it seems to break it like prior to adding the time limit.

Log show this error

Call to undefined function cdr_ob_clean_all() in /var/www/html/admin/modules/cdr/functions.inc.php on line 160,

Update:

My mistake went a bit too far on the function selection!!


You probably revoed the

mickecarlsson's picture

You probably removed the function when pasting in the fix. That function code was after after the replacement code.


__________________

Mikael Carlsson
(I am off-line, tinkering with my Chevy and my radios, don't know when I will be back)


Works great got 62,000 +

Snisk's picture

Works great got 62,000 + results in <30. I will keep testing. Any Possibility of adding a find all results to this?


Something is causing a

Snisk's picture

Something is causing a column shift with Lastapp and Lastdata Duration, Billable sec and proceeding columns are off. Looking into it.

Update:
lastapp column is shifting on us. Also clid is flowing over into src on rows with long clid's. Resulting in additional column shifts on certain rows.


OK, I did some more testing

mickecarlsson's picture

OK, I did some more testing and found where php suffers when allocation data.
This is time consuming in php:

$data = $data . $csv_line[$i] . "," ;
$data = $data . "\n";

Execution time: 43.395808935165 seconds

By replacing that with:

$data .= $csv_line[$i] . "," ;
$data .= "\n";

Execution time: 0.1939640045166 seconds
The speedup in php is enormous.

So, restore the functions.inc.php, then change according to this:

Index: functions.inc.php
===================================================================
--- functions.inc.php   (revision 12476)
+++ functions.inc.php   (working copy)
@@ -206,12 +206,12 @@
                                $csv_line[$i] = "\"" . $csv_line[$i] . "\"";
                        }
                        if ($i != count($csv_line) - 1) {
-                               $data = $data . $csv_line[$i] . ",";
+                               $data .= $csv_line[$i] . ",";
                        } else {
-                               $data = $data . $csv_line[$i];
+                               $data .= $csv_line[$i];
                        }
                }
-               $data = $data . "\n";
+               $data .= "\n";
                unset($csv_line);
        }
        cdr_download($data, $fname);

That will speed up execution, it should produce the same output as before.

As for download all, if you to download 100,000 records the memory in apache/php will run out.
I will look into that to see what I can do.


__________________

Mikael Carlsson
(I am off-line, tinkering with my Chevy and my radios, don't know when I will be back)


Okay updated changes. Looks

Snisk's picture

Okay updated changes. Looks good everything matches up. I still have a few where half the clid goes into src. Further research shows that the clid has a "," in the name field. Thus when the function to separate the csv file is done it thinks it should be a new column.

Example
,


OK, could you extract that

mickecarlsson's picture

OK, could you extract that line manually from mysql and pm to me so that I can see how it look like.

CDR should enclose all strings with a ',' with '"' during process of the csv.
Instead of open it in Excel/Open Office, save it to a file, then open it in Notepad++ (shameless plugin). Look for the faulty line and see how it look like.


__________________

Mikael Carlsson
(I am off-line, tinkering with my Chevy and my radios, don't know when I will be back)


Chan. Column info

newranman's picture

After doing a search in the Return Area.

Is it possible to show more info for Src Chan. or Dst Chan. like SIP/Ext# or Sip/Trunk without hovering over them. Currently mine says SIP unless I hover over it. For incoming calls I use ring groups and the Dest. says the ring group but the Dst Chan. is really the area I'm interested in it shows the ext the pickups the call in the ring group.

Thanks

Ran


There are limited space on

mickecarlsson's picture

There are limited space on the screen to have that displayed, compare to the old report tool, that displayed it, but it was cut off.


__________________

Mikael Carlsson
(I am off-line, tinkering with my Chevy and my radios, don't know when I will be back)


User interface tweak

newranman's picture

In the Call Detail Record Search

The form option buttons are on the left for "Order By" and "Extra options" but the search conditions have them on the right it is confusing, at least to my brain I keep selecting the wrong option. I suggest putting them all on the left.

Thanks

Ran


Fixed. It was confusing.

mickecarlsson's picture

Fixed. It was confusing.


__________________

Mikael Carlsson
(I am off-line, tinkering with my Chevy and my radios, don't know when I will be back)


My dream query request.

newranman's picture

I may be missing it but I don't see a way to show the total of call times. Say I want to know how many minutes ext 510 was on for today. Or multiple extensions in one report

The Call Graph by Day appears to do this I could run it for one extension twice. Once for incoming and one for outgoing then add them manually together. Repeat for all the extensions needed.

This is what I'd love to have I'd even be glad to donate some money.

1. Todays Outgoing total call count for a select "channel" or channels with the sum of the "billsec" for each "channel"
2. Todays Incoming total call count for a select "dstchannel" or dstchannels with the sum of the "billsec" for each "dstchannel"
3. Todays Sum of "incoming call count" and "outgoing call count" for Extension with the sum of outgoing_billsec and incoming_billsec

Example csv report output to one file:
-----------
#Outgoing
Extension, Call count, Billable Sec, Billable Minutes
500, 20, 375, 00:06:25
501
502
503
504
505

#Incoming
Extension, Call count, Billable Sec, Billable Minutes
500, 10, 180, 00:03:00
501
502
503
504
505

#Total
Extension, Call count, Billable Sec, Billable Minutes
500, 30, 555, 00:09:25
501
502
503
504
505

-----------

Thanks

Ran


Hi everyone, I just

rjefferis's picture

Hi everyone, I just installed this on my box and seems ok so far so thanks for all the work.

We use gsm for the recording format which i guess is still not supported as the player doesnt seem to work.

Could we just have a link to download the recording instead?

Thanks

Rob


Is there a more robust

ou812's picture

Is there a more robust version available like a pro version at $$$

Gary.


No call recording link?

mlungu's picture

Hi

I have been away for a while and trying to catch up with things, hence the delayed response here. The latest version has taken away the call recording link on my installation? Any ideas where and what to check?

Thanks a lot.


To Add 'Save' option to recordings

wvt's picture

Very clunky and maybe not cleanest way, but works for our purposes..

To add a 'Save' option for recordings...

edit cdr_play.php

Replace:

	if (isset($file)) {
		echo("<embed width='100%' type='audio/basic' src='modules/cdr/cdr_audio.php?cdr_file=" .$file. "' width=300, height=25 autoplay=true loop=false></embed><br>");
	}

With:

        if (isset($file)) {
                echo("<table><tr><td width=100%><embed width='100%' type='audio/basic' src='modules/cdr/cdr_audio.php?cdr_file=" .$file. "' width=300, height=25 autoplay=true loop=false></embed></td><td width=8></td><td><a href='modules/cdr/cdr_audio.php?cdr_file=" .$file. "' style='text-decoration: none;'>Save</a></td></tr></table>");
        }