Dimensioning an Asterisk system

Typical questions asked on the mailing asterisk-users are:

How fast/big must my machine be in order to serve my needs?
How many simultaneous calls can Asterisk handle?

The only rule of thumb we appear to be able to provide is this: Asterisk 1.2 start to run into problems around 220 concurrent SIP calls. Asterisk 1.4 scales much better and can handle nearly double the call setups/second as well as total concurrent traffic. Moreover, Early testing of Asterisk 1.6 using hash tables shows a SIP performance increase compared to 1.4 of factor 3 to 4. In asterisk 10 the performance increase in sip udp handling is around 2 to 3 times faster compared to Asterisk 1.8.

Apart from this, unfortunately, there are no simple answers. You’ll need work through the following checklist to at least get nearer to an answer or be able to post a meaningful question to asterisk-users.

1. What type of phones do you plan to use (analog, SIP, Skinny, H.323, MGCP)?
2. How many phones will you operate?
3. How many external lines do you have, and of which type (analog, BRI, PRI, T1, VoIP)?
4. How many concurrent internal/external calls do you expect (example: 30%)? Consult the Erlang tables (see links below) if in doubt.
5. What codecs will be employed, and will you need to do a lot of transcoding? Tip: Enter SHOW TRANSLATION at the CLI.
6. What features shall the system provide (echo cancellation, voicemail, conferencing queues & call center, recording, fax, voice menu, text-to-speech, speech recognition)?
7. How reliable/scalable must your system be?
8. How many Asterisk boxes do you plan to put in place?
9. What does your IP network look like (speed, QoS support, VLAN, Power-over-Ethernet)?

The next steps are to a) check out the hardware examples (use cases) section of this Wiki, and b) after that possibly set up a test system that is able to simulate your requirements so that you can run a load test e.g. using .call files generated by a script.

How many concurrent calls?

Here a collection of statements voiced on asterisk-users:

Note: When adding to this list, please provide date, CPU & Memory usage, not load average. These are much more concrete metrics and have much greater relevance when comparing different hardware and operating systems. Entries are sorted by CPU size.

Small and slow

• for old & slow CPUs take a look at Asterisk hardware recommendations
• Gumstix-based SIP-to-IAX proxy server can route about 40 calls concurrently (no transcoding, Gumstix 400XM, voicemail on CF card)
• Pentium 133 MHz, 16 megs of ram: Handles up to 3 concurrent SIP calls before quality degrades (no info about transcoding)
• Linksys NSLU2 aka “slug” (NAS Device for $79): Manages around 4 SIP calls with g711 or gsm (but not g729) on its IXP400 processor
• Pentium 1, 166mhz, 32meg ram: Successfully runs 4 SIP calls with codec g711
• Pentium II 233/64 RAM: 2xBRI (=4 ISDN channels) plus a lot of SIP devices
• The SC1100 CPU on the Net4801 will only successfully encode two streams of voice using G.729a compression. It is generally accepted that Asterisk requires about 30 MHz of CPU power per active voice channel. Thus the 266 MHz CPU on the Net4801, in theory, supports around eight simultaneous calls. This presumes that all calls are G.711 encoded audio. The Soekris Net4801, while inexpensive, has been absolutely reliable as a host platform, delivering adequate processing power to support six extensions and up to four active lines at one time. It could perhaps do more, but such was beyond the scope of my ability to test.
• (Mai 05) on a Soekris board: purely translating from G.729 IAX2 <-> G.729 SIP (maybe using IAX trunking), note that’s not a codec translation: I have people doing at least ~30 calls in that manner (using AstLinux)
• A Soekris 4801 can easily support 20 – 25 SIP clients if they are all running the same codec (I use ulaw), I know of others that have 20+ sip clients and a t-1 card in the soekris for zap channels and it works fine. If you have to do any sort of transcoding a soekris is not the way to go but for a small installation, it works great.
• Soekris 4801 at 266 MHz: Four concurrent ZAP calls might be tricky: even if you aren’t doing transcoding, you will still probably have to do echo cancellation, which is CPU intensive (especially on the Soekris)! As far as 60 calls: maybe SIP to SIP, with re-invites and even that is pushing it.
• (Nov 07) I have successfully compiled and installed Asterisk on an Alix board (AMD Geode 500 Mhz + 256 Mb RAM) on top of Voyage Linux (a Debian variant). I wondered how much it could be loaded, so I tested it with pbx-test: I could place up to 15 simultaneous SIP calls before it got no more responsive.
• Capacity of MeetMe: With 28 persons on a Pentium II, 300MHz, 128 MB vmstat shows 70% idle.
• 5 calls to music on hold (where it’s transcoding from the GSM moh file to G711 is causing my 533MHz VIA processor with 64Kb cache is using between 5 and 12% CPU.
• “I have anywhere from 15, to a peak max of 30 traders all using the same meetme conf during the day. My * is running on an old 4U 500Mhz machine (dual board, one processor installed now). With the exception of a few problems from software sip phones, our implementation has been relatively problem free.”
• VIA embedded 800 MHz: Max. concurrent and transcoded calls with speex: 4, ilbc: 9; g729: 11, g726: 22, lpc10: 24, gsm: 25, A-law: 83

Breaking the 1 GHz barrier

• The 1 GHz CPU (HP T5700) has managed 4 calls using G729 whereas the Soekris Net4801, with its 266 MHz CPU could only handle two…barely.
• (Feb 09) Claim: A 1GHz Via processor with 128KB cache will handle 100 concurrent calls with no transcoding.
• VIA EPIA 1GHz fanless: Compressed codecs are fine – as long as you aren’t transcoding 😉 I figured I could push 30 non transcoded calls through one, but I’ve never had the ability to fully test it out. The max. I had going on one system was 20 calls. This box has 256MB of RAM. Asterisk 1.2.16, zaptel 1.2.15. It’s running apache+php, sendmail, and asterisk. Boot it off flash and have it load an initrd.gz into RAM. Everything will run entirely from RAM – no writes to the flash at all! I can get everything inside a 48MB flash drive
• Asterisk 1.2 beta managed pretty well at SIPit: I ran Asterisk on a 1 GHz laptop and could easily handle 250 concurrent calls with re-invited RTP, That took about 9% of the CPU. With RTP codec conversion (ulaw-alaw, easy load) my poor laptop run at 99% load already around 50 calls…
• Celeron 1GHz with 2 Gigs of RAM that handles 72 DIDs, 24 connections to our traditional PBX, and is our incoming FX server. (Also running 1.0.9)
• July 08: 1GHz VIA processor, 128KB cache, OSLEC software echo cancellation can do the following: (running their own speed test program): Testing OSLEC with 128 taps (16 ms tail), CPU executes 996.06 MIPS; Method 3: samples clock cycles for each call, IIR average, 29.15 instances possible at 100% CPU load. So at worst, it’s saying it can handle 29 call incarnations, and at best, 37 – that’s assuming no other CPU load such as transcoding. On my dev box, an older 2GHz Celeron, 128KB cache, it’s telling me it can do 120 incarnations, and on a 2.4GHz Xeon with 4MB cache, it said it could do 321. Note: Those numbers are obtained with a 16ms tail, which is very short, and unlikely to be an adequate echo tail for connection to the PSTN (although fine for analog phones). A more normal configuration would be 32, 64 or 128-millisecond tails.
• Dual P3 1.13ghz, with 5 PSTN Channels via PRI -> G729 channels, has a constant load of 0.45
• I do have a dual 1.266 PIII with 2 GB of RAM that handles 75-85 concurrent SIP (GSM) sessions + 3 IAX2 trunks for a total of about 100 calls at the same time (using * 1.0.9). I also have this server set up for a file server (authentication using LDAP in a Win2K ADS domain).
• Intel P4 1.7G 512mg 20 gig HDD with 2 T400P boards for access bank and 60 analog phones; 24 channel PRI
• (Apr ’03) chan_h323: Takes advantage of the codec pass-through feature of Asterisk (unlike chan_oh323), but has no jitter buffer. We simulated 125 concurrent calls set up in random (sub-15 second) intervals, using passthru, on a 500mhz P4 without causing more than a 5-second delay in call setup time. On that same machine we simulated 45 concurrent calls using GSM transcoding, again without a long delay in call setup time. We have also a few production systems (single P4 2.53 ghz, 2gig ram) in pass-thru mode with 3 T-1s of concurrent real traffic.
• Today, we were able to reach an astonishing 790 simultaneous audio playbacks. (sip to asterisk, any codec.) on a 350$ pc. Tests were done on we a pIV on 2.4ghz with 512mb ram. Unless you dont do cpu hungry tasks such as transcoding you can go as low as 1% idle cpu cycles without the slightest audio glitch in this test
• Celeron 2.4 GHz, 386 MB RAM, 1xT1 for (only) 10 phones
• 2.6 GHz Pentium4 800MHz bus w/ HyperThreading enabled, 2 GB RAM (however 1 GB really is sufficient): 40 concurrent SIP-to-ZAP calls (maybe more are possible): “Measured for about a 12 hour period and over 5000 total phone calls per day. With this server you can see the load average jump from 0.00 to 6.25 in a matter of a minute”
• g729 performance taken from the Digium web site: Internal testing with dual Intel Xeon 1.8GHz processors allowed 60 concurrent calls. Dual Xeon 2.8GHz processors allowed 80 concurrent calls.
• “The load is extremely dependent upon what you are doing with it. For example, a simple IVR/Zap-T1-channels-only system can handle 10 times the number of consecutive calls of a SIP&Zap conference call system (at least in my experience).”
• MeetMe configs on single P4 (512M and 2G). I’ve seen no technical problems up to and over 30 attendees, all coming in via H.323 (chan_h323), using g.711u (no transcoding). Word of warning- above 10 people, a single misconfigured softphone attendee can create chaos. Based on load averages (at 25 people, single P4 with 2G ram and nothing else going on on the box, we see about a 0.5 load average), there’s no reason why it wouldn’t support 50 or 60 attendees before seeing problems.
• Intel Celeron DC E3400 2.60GHz Dual-Core, 2GB DDR2-800, running FreePBX Distro handling 30 concurrent SIP/ulaw calls with recording on two queues. Also running FOP2 pro.

Pedal to the metal: 3 GHz and above, SMP and dual/quattro core

• P4-3Ghz/1Gb RAM: Native format_mp3 for MoH running on FreeBSD 5.4 with Asterisk CVS HEAD (July 2005). mp3 decode quality is great… up to 70 simultaneous MoH (8khz 16bit mono mp3) calls with about 80% idle.
• (Aug 05) Pentium 4 (No HT) 3.0Ghz, 1 GB Ram. Handles 46 G.711 SIP calls while playing SayDigits(1234567890) to them all. Load average 0.43.
• (Jun 08) Our Asterisk 1.4.2 server can handle about 100 simultaneous calls having 3Ghz Dual Intl-Xeon Processor with 2GB of ram using G711 codec, and around 30 simultaneous calls using G729 codec.
• We’ve been doing some benchmark experiments on a 3GHz HT box with 1GB of ram, mirrored traditional IDE disks. The box has a Digium quad-PRI, a TDM40B, a TDM22B and a Sirrix quad-BRI board in it. This box can run 120 active calls over 4 PRI spans. Its running MusicOnHold into 60 of the channels, playing various GSM prompts into the other 60. The “user” cpu usage is about 25%, the “system” cpu about 25% also. (I imagine the system cpu will go down if I remove the “extra” boards). I can add to that 5000 registered SIP peers and 5000 registered IAX2 peers – total of about 100 registration refreshes per second. That adds about 40% more user CPU and pretty much fills up CPU. Audio quality is still perfectly fine though, and PRI slips few and far between. Load average for the whole mix sits between 5 and 10. About 550Kb/s out and 400Kb/s out on the ethernet for the registration traffic.
• 3.2 Gig Pentium 4, HyperThreading (HT) turned off, Memory 1 Gig, no hard drive, running in Ramdrive, Memory usage never goes over 256Meg. I’m using Asterisk with looping call test configs to play audio and using 3 of the same spec servers to pound calls through 1 server. I managed to get 350 concurrent calls through (g711, no transcoding) with perfect audio consistently with ~20% idle processor load. Anything above that and things start breaking up. Using Asterisk 1.2.6 I’m running into a limit of 276 SIP calls and no more. IAX calls can go 400+, so I test with combination 200+ SIP calls and the rest IAX and a combination of more and less SIP and IAX calls. With HT turned on, SMB loaded in the kernel gave ~20% performance increase, BUT, using 425+ channels gave very inconsistent results, choppy audio, calls dropped, no audio, and call setup time slowed.
• “Dual Xeon 3.06, 120 channels (60 calls) with H.323, G.729 using less than 10% CPU, no transcoding is being performed”
• We installed a quad xeon 3ghz which transcoded ~100 active channels (as a gateway). Take a look at the codec demands (in asterisk show codecs I believe) and scale from there. This box was 60% loaded – which is all we’re comfortable with before latency goes too high.
• “Look for the recent ‘capacity testing’ thread. We’ve had some discussions on it, but so far the bottom line sounds like you won’t be able to run more than 20-25 decent quality calls before asterisk dies when transcoding and H323 are involved.” (note: maybe this applies to oh323? For h323 the tests shown at Astertest indicate that for high CPU codecs like g729, ilbc and speex the protocol doesn’t matter, so h323, sip and iax should all perform the same here)
• “I read somewhere that one of our associates here has experienced about 45 calls when transcoding. The question becomes if this is all we can do with one server, what is the point of getting 4 E1s Digium card while one can never be able to use 120 channels transcoding from VOIP to TDM?”
• “My rule of thumb has been 100 G.729 channels for a dual 3.0ghz Xeon machine. Cost: around $42 per channel (I buy SCSI systems with RAID, and that includes the $10 charge for the G.729 license.) (Not including Digium 4 T1/E1 card) Your mileage may vary in both performance and price. Cost for an AS5300-style solution: around $110 per channel, and that’s on the used market pricing plan.”
• On my own testing I could get 200 simultaneous calls on a Xeon 2.4 using SIPP, each call was 10sec so the box had 20 incoming cps, obviously, this was using SIP not Zap. Since I don’t have another Xeon to put on my box how do MP scale ? I believe that I cannot get 400 calls for 2 Xeon but maybe 350.
• (Jan ’06) Dual Xeon 2.8 GHz Intel Chassis with 1GB RAM: Running +-35 IAX2 channels out a 2 x PRI (E1) Digium Zap channels. Importantly the box is recording all the calls and logging all cdr’s in a mysql DB.
• (Apr ’05) Dell PowerEdge dual 3.2 GHZ XEON (32 bit), 2GB ram: It’s configured using realtime-extensions / sipfriends / iaxfriends with a local mysql daemon, 80% of all calls are IAX <-> SIP calls with no codec transcoding and no jitterbuffering, and 20% of all calls are IAX <-> IAX native transferred calls, again no codec translation. The box has no zap channels. Now, when I have a load of 100 simultaneous users calling (that means I have 200 simultaneous lines open of IAX and SIP calls (where 80% of those calls are IAX calls translated to SIP/RTP) then asterisk uses an average of about 30% to 35% processor time (divided over the processors) and mysqld then using about 1% average load .. so the maximum callers I can have is about 300 ..
• One very important thing to remember when you are talking about SIP-to-Zap calls in Asterisk is the processor spikes. Taken on average, a quad processor machine might be able to handle 200 concurrent Sip-to-Zap conversations, but the reality is that there is a 4x load spike that occurs at random times with Sip-to-Zap conversations that would kill your big box when one occurs.
• An Asterisk system can only handle a max. of 250 concurrent ZAP channels. This is due to the design limit (255) within the ZAP channel driver. __Update Sep 2005: That limit has been removed a LONG time ago. Zaptel provides another interface for opening device files past the ~250 or so that exist in /dev/zap.
• Mark Spencer on Mon, 2004-11-29: I’ve got IAXTel moved over to “realtime” but it’s still very overloaded apparently causing long delays. There are currently 6000 registered users, over 1000 of which are typically actually registered to the iaxtel box at one time.
• (Feb 08) We have multiple installs that tested-out at nearly concurrent 400 SIP channels on a Dell 2950 with 2 x quad core at 1.6 Ghz, 16 GB of RAM.
• (Jun 08) Sangoma explains why with current Zaptel only a maximum of 500 TDM calls (two 8xE1) are possible on a single Asterisk box; note that in order to reach this number of concurrent calls hardware echo cancellation needs to be employed in order to drastically reduce the number of required hardware interrupts (video part 1, more on YouTube). Better – and distributed – scaling is possible using chan_woomera, according to Sangoma. YATE, unlike Asterisk, uses 1 interrupt per TDM span, not per TDM channel, resulting in a rough maximum of 30xE1 concurrent calls.
• (Oct 06) I am doing testing with SIPP tool. Now the thing is that asterisk is not able to handle more than 550 concurrent calls with call rate 100 call/sec (on my amd 64bit dual core machine with 2 gb ram). I am doing this with ‘canreinvite=yes’ mode so there is no issue of RTP traffic. I just want to establish peer to peer call thru asterisk (I hang up calls after some time so this is including all invite and hangup related massages)
• (May 05) In my testing of CVS-HEAD I can get 5551+ sip “sessions” without media on asterisk without a problem (testing with only 1 SIP user). The load average is around 2-3. On a side note… on my 3ghz P4 HT box I can get 629 ulaw sip calls with media (verified) without a problem. The load average on the box was around 14 and it still sounded perfect… so if you had a dual 3.4 ghz Xeon box you should have ZERO problems doing a DS3 with Asterisk. (That is if the interrupt is 1000 per second and not 28000 and its all ulaw) … note that if you do not set the ulimit -n 100000 or something similar before you start asterisk you’ll run out of FD’s around 151 calls.
• (May 05) But the real scalability wall I’ve seen is number of registered peers… That’s what takes down a box (at least with IAX). I’ve heard reports of a Dual Xeon 3.2GHz not being able to handle even 1000 IAX peers. … I managed to get about 2500 users online on one box by modifying iax2.h (reg expire changed from 60 to 240 seconds) decent IAX clients will comply with that setting .. b.t.w. don’t go higher, because many NAT gateways will close their dynamic NAT mappings after 300 secs, a few some even after 30 secs!
• (Oct 08) Asterisk 1.4.21.2, RHEL kernel 2.6.18-53.el5, Athlon 64X2 Dual Core 2.1 GHz, 1 GB RAM, Test tool: SIPP with pcap was used for generating pcma RTP traffic: 175 simultaneous calls listening to voicemail recordings, 100 callers recording messages. Note: One of the biggest contributing factors to Voicemail performance (and call recording/playback in general) is the details of the storage backend. This information has been omitted for your report.
• (Oct 08) PowerEdge 840 with a Quad Core Xeon X3220, 2x4M cache, 2.40 GHz, 1066 MHz FSB and 4 GB RAM. Redhat V5 was the operating system. The test was configured to simulate a wholesale VoIP operation with three-minute call durations and an average of two call retries for every completed call. This was an “out of the box” Asterisk 1.4.21.2 configuration with default settings and no optimizations. We found that Asterisk on the test server could handle approximately 1000 simultaneous calls with no codec translation. This works out to be about a $1 per port investment for a B2BUA platform. When calls were transcoded from G.711 to G.729, the call capacity fell to 320 simultaneous calls. With the added cost of the G.729 codec royalty and the lower call capacity, the cost increases to approximately $13.50 per port. Details here. For a server hosting Asterisk, each One GHz of CPU processing capacity can manage 100 simultaneous calls without codec translation (G711 ulaw) or 30 simultaneous calls with G711 ulaw to G729 codec translation.
• (Nov 07) We recently performed an in-depth performance test on Asterisk v1.4.11 configured as a SIP B2BUA. Asterisk was running on a server with two Xeon 5140, dual-core, 2.33 GHz CPUs and 4 GB of RAM. We found that an Asterisk B2BUA on this hardware can manage 1500 simultaneous calls with no transcoding (at a cost of $2 per port) and 400 simultaneous calls with G.711 to G.729 transcoding (at a cost of $17.50 per port incl. g729 licenses). A summary of the test is available, as well as the test details. In this scenario Asterisk, on average, attempts the call to two failure destinations before completing the call to the SIPp server.
• (Jan 08) I have a client with a four-core Xeon box doing SIP to IAX conversion – that box can handle 1000 concurrent calls.
• (Aug 08) I have set up a system with 180 users in meetme rooms on a single server (4xdual core Xeons) using a Sangoma a108D (8 port T1/E1 card with hardware EC with 8 x T1s connected) and the machine was running at high load but it was usable with good audio.
• (Aug 08) I have a Core2Quad 2.4G machine with one Sangoma quad-T card (without HWEC), that’s meetme-ing 72 Zap channels from Channelbanks into an equivalent number of IAX channels over 100BaseT; each conference has at least one extra Record() app running full time, as well as one playback. This is a production machine, but it takes careful tuning to keep it production-stable at that load (where my definition of production-stable is “if the load average breaks 2.0 for more than 5 consecutive half-minutes, I get nervous.”)
• (May 08) 2 x Quad-Core Xeon 5300: 2000 channels and 1500 concurrent calls are reported for MOR with Asterisk 1.4.15 on Debian 4.0
• (May 05) While you may not choose to put 10k users on Asterisk, I have. Many more, as a matter of fact. Some of these systems were simply media/application servers, while some handled registrations as well. While I agree that Asterisk needs some help on registration volumes and scaling, I’d not sell it short so quickly. At the moment, the only reason I still would use SER would be for the registration and call processing/load balance speed – Asterisk provides all that I need for back-end call processing.
• (Apr 05) Signate Telephony Server 5000 s 51 Gigabits per second I/O capacity sustains more than 5,000 Session Initiation Protocol (SIP) call streams per module using 80% of the capacity of a gigabit network. Up to eight uniquely scalable modules can share a 6.4GB/second interconnect based on SGI technology that enables a coordinated system supporting over 40,000 simultaneous call streams. (Signate terminated its business in 2006)
• (2011) I had done some load tests with Asterisk 10 and my highest results were: 1750 calls per seconds (cps) up to 13000 concurrent calls done on a intel xeon with dual six core and hyperthreading (= 24 cores) and 12 GB ram. the sysload was around 2.5 during this test (with open rtp ports but without much

media transported, only signaling). With MusicOnHold I had to stop testing at 10000 concurrent calls due to the NIC being limited to 100 Mbit/s.

Providers

VON magazine Dec. 2005: For 2004 Voicepulse announced having run 10 million minutes of IAX traffic. In Dec. 2005 that is about the monthly amount of their IAX traffic. An Asterisk-based ITSP in Brasil claims 100.000 users.

Call centers

• (Nov 05) We use boxes for outbound dial setup. I don’t think I’d attempt to go past 4 E1s (120 lines) which would be 5 T1s. If the box is running hard like that the load average will sit around 7 or so, still fair amount of spare CPU but there is no way an Asterisk box will run well with the CPU anything like maxed out. Our site has 250 agents or so and the work is currently spread over 6 servers with 3 E1 PRIs on each. Each box makes around 3000 to 5000 call attempts per hour.

CPU load graphs

On this link http://quasar.sourceforge.net/bench.htm you can find the article from which you can obtain CPU loading depending on a number of immediate zap-to-sip calls information.
Lessons learned:

• with zaptel software echo cancellation switched on the number of calls a box can take is cut in half (measured with transcoding SIP <–> E1)
• without echo cancellation up to 240 concurrent SIP-to-E1 calls are estimated on a 3 GHz Debian system with 2.6 kernel

Determining limits

What really matters is the load on your system and the type of channels you are using. Zap channels are more reliable at extremely high loads than any VOIP channel. Manager Actions can fail at extremely high loads. All VOIP channels will suffer quality loss at extremely high loads. By extremely high loads I mean about 5.00 and higher of system load (10.00 on dual CPU machine) There are other factors, but this has proven fairly reliable for us. System Load by channels in use is different from server to server depending on the CPU/RAM/Motherboard and other application running on the server.

Digium Hardware (July 2005)

Digium cards are now smarter! Digium is proud to announce second-generation firmware for Digium multi-span TE-series hardware PCI cards. This firmware provides several new features. These features allow Digium cards to do more work in hardware and leaving the server’s CPU open to do other tasks, all of which contribute to less CPU overhead, and therefore improved performance, and more channels running through a single PC or server.
… This is a 67% increase over previous benchmarks …

The new firmware solution relies less on the server’s CPU for the operation of the Digium card, therefore reducing CPU overhead and improving performance and increasing channel processing. For example, a dual-processor, 3-GHz 800FSB Intel XEON server with 1MB L2 cache, and a Digium 4-port T1/E1 card, can now convert 120 SIP channels with G.729 compression to the PSTN without Digium’s echo cancellation module and 150 channels with G.729 compression with the Digium echo cancellation module.

If you have very high traffic volume

Scott Stingel www.evtmedia.com writes:
My largest client has a configuration (15 asterisk boxes) that takes up to 1800 simultaneous IVR calls, supplied by a large private DMS-100 PBX over 60 E1 spans. The calls typically are very short (5 seconds).

• I suggest: Use 1 processor (chassis) (example: 2.8Ghz P4) for every 4 E1’s (example: one TE405P card) when you have a large amount of call setup traffic. I tried a dual-Xeon to see if I could handle two TE410P’s instead of one in a pure IVR (Zap) environment, but many calls were dropped under full load. So I’ve now decided just to put one TE410 or TE405 in a box as a maximum. If you are doing a limited amount of transcoding, and don’t have a lot of call setup load, you might get more than 120 channels in a system, but be cautious! Also, I haven’t re-tested using the new Digium firmware, see above)
• Try to do as much as possible using the dialplan (extensions.conf). AGI scripts are very powerful but cost you in performance when you’re running large numbers of lines. (more info, see: designers’ FAQ #2)
• For the processor to use, see the Wiki for suggestions. I’ve used P4 and Xeon based Tyan and Intel motherboards with success.

512 simultaneous SIP-to-SIP calls with Digital Recording

How to use a RAM disk to eliminate the I/O bottleneck associated with digitally recording calls via the Monitor application
Hardware for Asterisk: Quad Xeon MP CPU at 3.16 GHz with 20 GB RAM (Dell Poweredge 6850)
The 512 calls was also the only load on the system during testing. We are actually using the box in production now and with the additional load of agents, announcements, MOH, unattended monitoring, NFS transfers of the leg files to a remote box for mixing, etc. it looks like a more realistic number is somewhere between 250-300 simultaneous calls. For instance, at this moment we have 45 calls in queue and 116 active calls (resulting in roughly 250 leg files being generated) and the box is about 65% idle. I’m assuming that the scaling will be somewhere between linear and exponential, which is where I’m deriving my projected numbers from.

• Follow-Up

Load testing – Zap

Scott Stingel: You can build Zap load testers for asterisk, using the outgoing call facility (see sample.call). What I’ve found:

(a) I don’t think it manages to queue very well if there are a limited amount of outbound resources. For example, if you define a group (“g9” for example) of two lines for use in outbound calling, it works fine if the number of outbound calls to be made at any moment never exceeds 2. A third call file in this example will be grabbed by Asterisk, but will fail immediately. So I had to create a mechanism in my Perl script to ensure that the outbound calls actually completed – no easy feat since I couldn’t tell when that occurs from the perl script too easily.

(b) There was a problem dumping more than about 12-15 outbound calls at once in the outgoing directory, even if there were plenty of channels available to make the calls. Asterisk would grab them but would not process some of them. This is a load-testing scenario, and not too common I realize, but something to be aware of. It didn’t seem to matter if I switched to a more powerful processor. Note that this problem may have been fixed (see below).

Jesse Janzer: Just a quick mention, we implemented a generic paging system and regularly dump more than 30 files into the outgoing spool (during an agi script) at once to accomplish the page, never had a problem…

The outgoing facility works fine at lower call volumes if you stagger the creation of the files in the outgoing directory.

Tip: You can manage when the call starts by setting the mtime of the blah.call file.

Quad T1 card performance testing – Digium vs. Sangoma

Load Testing – IP

• Juanjo: Remember to increase the max number of file descriptors and RTP ports.
• Sipp is a great load testing tool

pbx-text (php)

Atis Lezdins (iq-labs.net) has written test framework: you’ll need another machine with Asterisk (+php) on it to generate calls. It allows writing scripts in PHP to emulate random customer actions, etc.

sniff2sipp

Terry: I have written a tool in perl that allows you to take a SIP pcap capture (live or saved dump) and generate a sipp scenario from it. It also should support recording/sending audio with the pcapplay option of sipp. It has been a little while since I worked on it and it isn’t particularly pretty, but if anyone is interested I could probably get it posted somewhere.

sniff2sipp can now be checked out through svn:

svn co http://svn.digium.com/svn/sniff2sipp/trunk sniff2sipp

Test1:

I have reached 1500 concurrent SIP/GSM calls against the Echo application at 50cps of 10s duration on a Xeon 2.4Ghz single cpu. Anyway, I m trying to make a more realistic scenario since I believe Echo application doesn’t t process audio too much just copies it.
When testing I monitor Asterisk against a 7960 calling MusicOnHold at about 500-600 concurrent calls I start to have audio dropouts.

Test2:

I set up a UAC which was generating 10cps of 10s duration and the corresponding UAS which received this calls. The command used to generate the calls which were GSM was:
sipp 192.168.65.100 -s 700 -sf uac.xml -d 10000 -r 10

The command to receive the calls on another box was:
sipp -sf uas.xml

I m using my own uac.xml and uas.xml just to talk GSM, I monitored using my 7960 against a MusicOnHold. On my Xeon 2.4Ghz no call was dropped and no audio problems. Note that
I use nat=yes and canreinvite=no for UAC/UAS in Asterisk config sip.conf. Older versions of SIPP didn’t support authentication.

• For 40cps of 10sec duration (which means 400 concurrent calls) it works just fine for me.
• At 50cps of 10sec duration, no call is dropped but I start seeing some SIP packets retransmissions.
• At 60cps lots of call gets dropped but the funny thing is that the audio through the 7960 isn’t much affected.

However: I’m not seeing any RTP traffic generated from SIPP UAC which means that * won’t generate any traffic at all since it recovers the clock from the received stream AFAIK. So maybe the tests aren’t measuring the whole * capacity just the SIP stack. One solution is to end the calls on a MusicOnHold extension but passing through an intermediate * which is the measured one.
A cleaner solution for Asterisk 1.2, however, is outlined here: Using SIPP to stress-test Asterisk. Finally, bug/patch 5374 should fix this for Asterisk 1.4 and later.

SIP test 3 (2006, Asterisk 1.2)

Asterisk’s 1.2 RTP stack requires bidirectional traffic to actually send traffic back. There was a patch in Mantis a while back that did asynchronous RTP, but I was unable to find it. However, the fact that you need to receive an RTP packet to SEND an RTP packet is the reason why no RTP was originating from the Asterisk server.

So here is what I did to get things working to create boatloads of 120-second calls to a Music On Hold extension:

1. Installed libpcap and libnet on my development box.
2. Built SIPP w/ Pcap Play support using “make pcapplay”
3. I grabbed a g729 RTP stream from a SNOM phone w/ TCPDUMP:

tcpdump -T rtp -vvv dst 207.166.192.106 -w g729.pcap

4. I then dumped the “uac_pcap” XML file and made some tweaks:

./sipp -sd uac_pcap > uac_pcap.xml

My edits consisted of modifying the following line:

<exec play_pcap="pcap/g711a.pcap"/>

to

<exec play_pcap="pcap/g729.pcap"/>

And, I changed:

a=rtpmap:0 PCMU/8000

to

a=rtpmap:18 G729/8000

5. I then executed the following:

./sipp -sf uac-g729.xml -d 10000 -s 451 gw4.n2net.net -l 96 -mi 207.166.192.254 -mp 5606 -i 207.166.192.254

And presto: LOTS of calls and LOTS of RTP traffic. Once you send one RTP packet to the Asterisk server, it responds back to the sipp RTP mirror port, and things just chug along after that. For results see: http://lists.digium.com/pipermail/asterisk-dev/2006-June/021166.html

SIP testing 4 (May 2007)

I’ve drawn three conclusions from this set of benchmarks (uLaw only, no transcoding, Asterisk business edition: ABE-B.1-3).
1. At low call volumes, the dual-core server outperforms the single-core server by the expected margin.
2. Calls bridged to an agent are more CPU intensive than calls listening to audio via the Playback() application or calls in queue. This is expected because they involve more SIP channels and more work is done on the RTP frames (bridging, recording, etc.).
3. For all call types, the majority of the CPU time is spent in the kernel (servicing system calls, etc.). I’ve observed this to be true at all call volumes on our production server, with the ratio sometimes in the range of 20 to 1. This may suggest that the popular perception that Asterisk doesn’t scale well because of its extensive use of linked lists doesn’t tell the whole story.

SIP testing Asterisk 1.6 with hash tables

• Early testing: 3-4 times load increase compared to Asterisk 1.4

Test environment: P4 2.8 GHz (Dell 32-bit PC), 1 GB of RAM running an old version of Ubuntu. U-Law only, no transcoding; all testing done with ‘ulimit -n 8196’ done previously. I have to do this, or I bump into the ‘too many open files’ problem. The threshold is about 195 simultaneous calls; 4600 sip friends in sip.conf.
For the short calls (0.1 seconds): ./sipp -sn uac 192.168.134.252 -s 12 -d 100 -l 256
For the long calls (100 seconds): ./sipp -sn uac 192.168.134.252 -s 12 -d 100000 -l 270

On standard Asterisk 1.6 trunk (Jan 2008):
Short Call test: At 100 calls/sec, I start seeing some retransmits after a while at 120 calls/sec, I see 0% cpu idle and sipp goes nuts.
Long Call test: At 270 simultaneous calls, cpu idle bobbles down to 0% frequently, but asterisk seems to hang on. At 280, things fall apart pretty swiftly.

On my turbocharged V8 version using astobj2 in team/murf/bug11210:
Short Call test: Things fall apart at 450 calls/sec. cpu idle drops to 0% and asterisk falls apart.
Long Call test: At 300 simultaneous calls, cpu hits 0% every now and then, but asterisk remains stable. At 310 simul. calls, we are stable for quite a while, but the cpu idle % times keeps bobbing up and down, and when it hits 0% hard, asterisk starts to trip up.

Conclusions: While I can get 4x performance for short calls, I seems to be dealing with some sort of network limit or something for long calls. Still, using hash tables to cut cpu usage, asterisk seems to be able to squeeze a few more calls in before it gets overloaded (about 10% or so?).

SIP testing 2010

I would like to share with you an article [1] we have issued last week (sorry, currently only in Romanian language – we plan to provide an English version soon).

This article is describing a method to be used for obtaining the maximum number of SIP simultaneous calls an Asterisk server could process safely (meaning no errors/maintain control of the machine and without RTP frame drops)

We used SIPP (with modified uas and uac_pcap scenarios) + 2 scripts for controlling the test (one is running on the tested Asterisk server – start-test.sh, for data collection and load analysis and the other is running on the SIPP+Asterisk testing machine, for call quality control and SIPP instance control – sipp-controller.sh) + customized Asterisk dialplans and SIP configuration.

The best part is that this method could be used for testing any type of Asterisk PBXs (from embedded to bigger servers), having capabilities to balance the load to several SIPP call generators/answer engines in case the tested server have more processing power than the testing machine. We have use this method to test 4 machines and the results are for the maximum number of G.711 ulaw – ulaw SIP calls are summarized in [2].

Also, this method is describing how to configure SIPP and Asterisk in order to test different transcoding scenarios (like ulaw to gsm).

Basically, the controller script increase the number of simultaneous calls (one SIPP call generator is calling an extension on the tested Asterisk server and the call is answered by another SIPP answer engine) till one of the load or quality tests failed.

The tests are:
– load evaluation -> how much time a `sleep 1` command takes on the
tested server
– SIP RTT evaluation -> what is the average RTT of a SIP INVITE message
– audio quality evaluation -> based on evaluating of the call “monitor” file size (on the tested Asterisk server we use an echo application and the file is recorded on the testing machine)

Even that the translation service provided free by Google is not the best way to read our article in English (or other languages) I encourage you to read it (the pictures and the results are very easy to understand) and send your feedback or comments here.

Best regards,
—
Ioan Indreias
www.modulo.ro

SIP testing 2013

Here our last testing in Dec. 2013, under Asterisk 11.2Cert.
Under Centos 6.4, Kernel 2.6.32-431.1.2.0.1.el6.x86_64, with double Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz, and with Asterisk in Realtime user, under Mysql with ODBC connector in the same box, after running from the remote box:

./sipp -sn uac -d 20000 -s 2005 tested-swith-ip -l 10000 -trace_err -m 20000 -r 100

Until 1600 simultaneous calls, the traffic was fine and media was correct with other bilateral calls, testing echo, and live calls… but after 1600 simultaneous calls, the media quality was highly degraded, and server become highly unreachable in ssh, and not comfortably responding. So, my conclusion here, under this hardware, I’d set the maximum capacity as 1600 calls, even, in the originating testing box, the sipp result was showing: call rate 59.527 cps

Regards,

www.sipcel.com

Testing Details & Notes:

• (1) Report in Romanian
• (2) A maximum number of G.711 ulaw – ulaw SIP calls:

38 – Norhtec MicroClient Jr DX
130 – VIA EPIA EN12000EG
176 – Asus Pundit R350
320 – Gigabyte 945GCM-S2L

Notes

• Transcoding is very resource hungry (CPU intensive); the same applies to echo cancellation; however, both tasks can be moved to hardware cards in order to relieve the CPU of the Asterisk box
• g729 codecs licenses can be purchased for Asterisk, whereas g723.1 licenses are not available and therefore Asterisk can only provide pass-thru functionality for g723.1
• IAX trunking can help to significantly reduce network traffic for 3 or more simultaneous calls between two Asterisk machines
• Hyperthreading is not well implemented in the 2.4.x kernels; the newer 2.6.x versions are more advisable to be used if HT is desired

Erlang tables

Calculating the number of external phone lines needed

• http://www.erlang.com/calculator/
• https://www.itu.int/itudoc/itu-d/dept/psp/ssb/planitu/plandoc/erlangt.html

See also

• Asterisk hardware recommendations
• Asterisk Appliances
• Asterisk rollout tips
• Asterisk at large
• Asterisk embedded systems: As small as it gets
• Asterisk installation
• SIPP load testing tool
• Asterisk performance and stability research (StarTrinity)

---

Go back to Asterisk hardware recommendations