provided by Star2Billing S.L.

Hello,

Very unfortunately I have had to discover that A2Billing is leaking calls.

Some calls are done but not stored in the mysql database, and hence not billed. The database is on a remote server.

The troubling line in the attached debug file is:
[10/09/2008 11:52:06]:[file:Class.RateEngine.php - line:851]:[CallerID:12345678]:[CN:5555555555]:[[CC_asterisk_stop 1.1: SQL: DONE : result=]]

Apparently A2Billing is happy to receive an empty result (nothing done) from the database. And so the call is lost.

As other calls are properly stored I can only guess that some temporary problem prevents occasionally the database queries. In any case the whole issue is disturbing as temporary connection problems can always happen.

To issues arise:
1. How to debug where the problem is and why these database queries are not done ?
2. Picking up the error when it happens. It appears to my humble mind that after each and every insert and update query of the database the result returned should be checked by the rating engine and if the database result is empty then the query should be written to a logfile of lost database queries. Then at least one can manually rerun all these queries.

Any input is highly appreciated since this problem is a sure money loser.

Cheers

Geejay

Hmm... I've been running several busy A2B systems on Postgres for years now without ever having seen this issue.
How many calls do you have active when this happens? Let me guess, just over 100 active calls right?

Stavros,

this actually very low load, under 10 calls. Asterisk machine and database run in under 10% load.

Perhaps you could run A2Billing in Debug mode and then

do

grep -B 1 "result=]" a2billing_agi.log

That shows you all queries which were not done, those with empty result.

How can I debug this further ? PHP is not throwing any errors on these missing queries.

The only persistent php error I get is about timezone missing. Apparently A2Billing uses some date() function that php doesnt like.

Any input would be helpful as this is driving me nuts.

Geejay

I checked back through 6 months of logfiles on 3 of my production systems before I replied earlier. I've never experienced this issue, and I don't think it's been reported here previously either. I'm running Postgres though rather than MySQL, which may be significant.
And MySQL's logs?
How remote? If it's not on the same Ethernet segment try re-testing with a more local SQL server.
No it doesn't. You need to correctly configure your PHP installation.

Stavros.


Not necessarily because both databases seem to be run through the adodb library. So symptoms should be the same.



The mysql server is standing right next to it.



I will check that thanks.

Stavros,

This server is in running in a virtual machine. Although the virtual machine does not show that any application tried to go over the limits I suspect now that certain memory parameters may kill some processes, or that some processes do not start because they see low memory. That might explain why there are no errors.

I have now increased memory for the virtual machine and crossed fingers.

In any case I think that A2billing would be well advised to check the result of the db query so one realises that certain calls were not logged.

Lets see what happens in the next 24 hours.

Thanks for your very helpful input.

Geejay

Yes the symptoms would be the same, but I'm suggesting that this might be a problem that only affects MySQL installations.
As the row is not inserted by A2B's query the SQL server should be throwing an error or warning of some sort. Did you consult MySQL's logs as I previously suggested?
You'd be getting OOM warnings on the host's syslog if that were the case.
Personally I would suspect the guest has CPU quotas or throttling rather than memory problems with only 10 active calls. As with all Asterisk installations, you should ensure the maximum number of file descriptors are available to Asterisk.
See /var/lib/asterisk/agi-bin/libs_a2billing/Class.RateEngine.php around line 901-902:Add this directly below:Then try running the logged queries at a MySQL prompt to see if they fail again.

Stavros,



Zilch, nada, nitchevo. Apparently the query never hits the server. There is nowhere any sort of error message, not in php, not in mysql, absolutely nowhere.

IMO the process just dies, hence the result is empty. Thats why I suggest that the A2Billing rating engine should always check the db result value and throw some errors. One cannot rely on some subprocesses to throw the error. The error must picked up by the highest process which supervises everything, which is the rating engine itself.



The guest has plenty of computing power. I have increased file descriptors now. However, the more parameters I change the less are the chances to find the real cause of the problem...



The queries run just fine.

Query logging as suggested is only partially helpfull for reconstructing the billing as the query contains CURRENT_TIMESTAMP and now(). So if one reruns the queries all timestamps of the calls will be wrong. IMHO the rating engine should ideally build the full query first with explicit dates and no timestamp variables so one can later rerun the queries from the log file.

BTW: The whole issue illustrates that one should ideally have some other switch which also rates the calls in parallel. Without it we would never have picked up that A2Billing leaked these calls.

Thanks again for your input.

Geejay

To my eyes, the whole issue illustrates, one more time, that running virtual machines instead of real servers is an unnecessary cause of trouble. Those machines may seem amazingly easy to setup, awesome to watch operate, BUT they keep being a hugely complicated layer between the hardware and what you intend to run on it.

Is the SQL server idle, apart from the A2B queries? What are connection_limit and log-error set to in my.cnf? Have you verified no connection is attempted using tcpdump or some other sniffer?
That's what you suggested, so that's what I implemented. As you say the logged queries run fine (excepting the timestamps) when resubmitted to MySQL, this really does start to point the finger of blame. How about rather than merely logging the failed query, if it persisted in submitting it twice per second until MySQL actions it properly or until 10 attempts have been made:

Yes, I couldn't agree more. It's always reassuring to have something against which to reconcile your figures before receiving the bill from your carrier, whether you're using GPL software or commercial. Asiby has been working on a billing daemon for A2B, hopefully to be seen in v1.4 one day, which should prevent any lost CDRs even if Asterisk crashes. Xrg has done some fine work on v200 where all the billing is handled by the SQL server itself, again having the side effect that a CDR can't be lost entirely.
If you're referring to VMWare or Xen or indeed most virtualisation techniques then yes you're quite correct. However in my experiences with OpenVZ and especially Linux-VServer the overheads are very minimal indeed: one kernel, one process list, one scheduler, just multiple kernel namespaces to load differing user space OSs into. They're more akin to BSD jails than virtual machines. Consult the latest VServer kernel patch. I suck at C and I've no kernel experience, but I have no problem reading that patch. It all looks very straightforward. It would be half the size if it didn't also implement COW hard-links for every in-kernel filesystem.

As Stavros mentioned Vserver and Openvz have very little overhead.

Also you have the ability to see where you are actually running short of resources. It would be much harder to pick up resource shortages when you run on the bare metal.

On top of his you get the following freebies:
- Ability to migrate servers to other hardware in case of hardware failure
- Ability to utilize underemployed hardware by running several VMs inside
- Full backup of the entire machine in a snapshot
- Ability to run production and test servers in parallel without additional hardware
- Ability to upgrade hardware by moving VMs to other hardware temporary
etc.

I venture to say that running VMs is even worth it if you are running only one VM on each physical machine, just because you are able to make full backups of your VM and be sure to be able to run those on any other hardware without any further reconfiguration.

Most important in our biz is that using VMs allows you maximize uptime without employing piles of idle hardware which is just sitting there. So you get max uptime at minimal costs.

Sorry, but you have not convinced me, still.

Ask yourself, why couldn't you do the above onto bare metal installations?

For example, why don't we have a nice and square way to move one a2b installation to a different machine by copying a few /etc/ files and the db ? Why can't we run anything alongside a2b on the same server (utilizing our free resources)?
Why do we need a full snapshot of every bit of the filesystem, for a backup?

On the other hand: are we sure that the VM will just work as if it were different hardware? That the heavy processes on one VM will not disrupt the sibling VM running an asterisk? That the asterisk will have truly stable timing?

IMHO, the challenge is to have software written well, and let it run alongside other software blocks on the same machine/OS.

Its not up to me to convince you. I can only suggest to give it a try on a small test installation..

Stavros,



Whatever the reason for the failure, for starters I would love to see some descriptive error message from A2Billing or any of the used libraries. Without such error message one can hunt for ages and still not be sure to have found the real cause.

Looking at the adodb documentation it appears to me that the error logging methods of adodb are not used in A2Billing.

[url]http://phplens.com/adodb/using.custom.error.handlers.and.pear_error.html
[/url]

http://adodblite.sourceforge.net/errorhandler.php

Where would I have to include this in A2B ?



I am very reluctant to include such code because it basically just masks the problem.

I want to find the cause of it all. I have increased VM resources and also increased file descriptors, still A2B is leaking some calls.

Thanks for your help

Geejay

Class.A2Billing.php
Of course! That code wasn't intended to be your final solution, merely to help you locate the real source of the problem: it may possibly prove that there's some transient problem outside of A2B's control.
You didn't answer my questions, so I'm still just guessing here.

Stavros,

the mysql server is servering queries from other servers with no problems. Simultaneous sonnections are all within specified limits. Also mysql doesnt throw any errors.

The point I tried to make was, that it is not good to change all sorts of parameters in all systems unless one understands what the cause is. The cause can only be found if A2B or some library throws an error.

Cheers

Geejay