provided by Star2Billing S.L.

I've been running what I would call moderate call volumes through A2B. What I've found is that the system load goes through the roof if I try anymore than 75 simulateneous call setups. CPU loading is around 30%. Not sure what's causing this, but Asterisk on its own without A2B seem able to go to 200 simulateneous calls quite well. I'm starting to think that I may well have to batch process CDR files.

Regards

How many cards/accounts do you have on the system. Just out of curiosity.

Hi

Do you have debugging set to 1 or 0 in the appropriate agi-conf? You get speedier results, once your platform is live if you have no debugging on.

Are you writing these logs to file ? That can impose a system load as well.


Is the database on the same machine as A2Billing, or a separate database?


Joe

To answer questions

100 accounts,

DB is local, MYSQL is not the issue as remote servers accessing the db for A2billing purposes are fine. Issue might be that Asterisk is being hit by around 50 call setups at once.

I'm seriously considering doing a mod on A2B to batch process calls after hours.

regards

One of the issues is MySQL. Out of the box, the maximum number of connections that you can have with mysql is 100. You need to change the mysql configuration file to fit your need. I have always boost mine to a couple of thousands connections. You don't want your MySQL server to be a bottleneck do you?

Cheers

Hi

I find it difficult to believe that the bottleneck is asterisk, not to that degree anyway.

The bottlenecks can only be:-

running the php script
writing to file - e.g. logs
MySQL

What version of Asterisk are you on?

Joe

I doubt the limit on DB connections is to blame: If you had hit that limit, you would get failures, not high load.

MySQL, asterisk and syslog are Linux processes. You could look at their cumulative time and judge if they had been spinning a lot.
Unfortunately, the PHP script is a short-lived process, which itself will not produce any useful statistics. **hmm, correction: ** there is "getrusage()" which you could append at the end of your script and have it log its execution time.

MYSQL is not local, so that rules out it being a resource hog locally. Put it this way, I now have 4 servers looking at 1 remote A2B database, and they all experience the same issue. I also tried it local, same result. It's possible that I need to space the call setups. It might be syslog, but my guess it's PHP. OS is Centos

regards

Middletn, when someone launch a "Denial Of Service" attack against your servers, you will notice that the CPU usage jumps to the roof, right? It doesn't mean that their are running a local process on the servers. The attackers will "always" do it from a remote location. In your case, what is the one common things that all of your servers share together? The remote MySQL. It is only logical not to dismiss MySQL as possible cause without doing a real scientific investigation with concrete proofs. I said earlier that it might be the maximum limit. In addition to that, it can also be due to the way MySQL is handling the connections being done over a TCP socket. If they are being slightly persistent, that the connection will stay open on the DB server longer than it will on the Asterisk server in the AGI script. Where is your DB server located? Is it on the same local subnet where your Asterisk servers are? If not, than "bad idea". That will lead to problems like "MySQL server has gone away" or "Connection timed out", ...

Here is one other example. Let's say that you are a web site running Joomla or Drupal and that the backend DB is located somewhere on a slow Internet connection. The overall speed of the web site will be greatly affected by the remote DB server sitting on the slow connection. I know that it's a silly example, but it was all I had in mind.

Nik, we obviously have no exact knowledge of your system but just trying our best to help you out with this. We're also trying to improve a2b in any way. Said that, bear another few guesses:

I have the feeling that CentOS has the best software around, assembled together in a really crap outcome. Things like SELinux, excess auditing etc. may make simple system calls perform lousy.

What is the *load* of those systems. Not the CPU one, the 1min,5min average ones. What is the swap ratios?

Do you use some php-eaccelerator?

Try to measure the network usage at your db server. Is the bandwidth, or *the packet rate* there high?
There is some lesser-known issue with TCP client-server applications: the small packet bottlneck. If you have a very bare protocol trying to communicate small commands (in the range of a few bytes) over a TCP link, it becomes very inefficient. TCP packets must be assembled and filled up, or /expire/ before they are sent. So, two ends that need to excange a lot of small TCP packets will end up waiting each other a lot. And that will only add up to the average load, no other indicator.

Ok guys,

Sorry I've not been providing enough detail and I know it's hard to make suggestions in a vacumn.

Setup is thus

I have 6 servers (A2B machines) connecting to a server (HP DL360 G5) running the database. They are connected to a Cisco 2950 switch and are the same subnets. The switch connects to another 2950 which routes to my PSTN termination provider some 30m away.

Each server generates calls with a delay of 0.5 seconds between each call. Each server has a min of 2gb of ram and the DB server has 6gb. Calls are generated using .call files (why? because I'm putting together an emergency broadcast and appointing system)

In asterisk.conf I have limits set such that if the load is greater than 0.9 x number of CPU's I reject calls.

All servers have centos 5 installed, apart from one which has RHEL 5 and asterisk 1.4 (latest version on all but one)

If I push more than 75 call setups through any server the load spikes above the max value in asterisk.conf. It's not waiting on interupts and iotop shows nothing excessive.

CPU usage never goes about 35% According to Top, there is no usage of the swap drive.

If I take A2B out of the equation, 200 calls per server are fine.

So I believe this is PHP related, but can't be sure.

If I use the server with the local DB as the dialer, same result.

No php accelerator is installed, but it might be worth a shot

Hope that gives some insight.


Regards

It is always your call to decide how much info to disclose. Thanks anyway.


In our next attempt (after the Linux issues), let's prepare to do net-stats from the cisco's info.


Well, that's interesting, this is a testing/pilot setup you mean? In that case, I would be dead curious to try v200 instead and see how it performs.


Subjectively, I would question that. I have seen Linux 2.6 perform quite well with load 4.0 on single-processor machines. You could set your limits higher.


/me rants about CenOS and old releases



I think that the number of processes spawn for the PHP etc. may contribute to the load. Plus the fact that w/o accelerator it needs to re-parse the PHP scripts at each call.

What is the memory allocated to PHP? Maybe that is what's being maxed out.

Cheers

128mb, should be plenty

regards