Voice of VOIPSA

“I think any time you expose vulnerabilities, it’s a good thing.” — Janet Reno

Welcome to the fifth installment of Asking The Cisco Systems IPICS Expert, security questions concerning the IPICS interoperability solution. As a reminder, these questions are all derived from my research and reading of IPICS documentation, as well as publicly available information on IPICS deployments. I have not had the opportunity to actually gain access to the IPICS to conduct security testing, though I look forward to the day that I do. If any folks out there are interested in providing me with this access, please contact me. Also, I can point you to several very skilled security engineers/researchers who can likely provide much deeper analysis and testing than I can.

So, for an update, the ipicsasktheexpert@cisco.com email address is still bouncing (screenshot here). Really, this has become a bit ridiculous considering the communication (albeit one-way) that has taken place between Cisco and myself. However, this snafu pales in comparison to what happened last week when a journalist from IDG contacted me about the blog posts. After a few emails between the journalist and myself, he contacted Cisco Public Relations and asked them what the story was with the bouncing email address and if they were aware of the blog posts asking IPICS security questions. The Cisco PR person apparently checked with the “IPICS team” and in turn told him that the ipicsasktheexpert@cisco.com email was working fine and that they have fielded many questions. Also, the Cisco PR person told him that Cisco has no record of my emails to ipicsasktheexpert@cisco.com. Finally, the Cisco PR person told him that the IPICS people at Cisco had only very recently become aware of my blog posts concerning the IPICS.

Rubbish.

I detest lies. And for Cisco PR to do this really undermines their credibility in this situation. I have the record of emails, including to top people in the IPICS team, from more than one month ago. I have the bounced emails from ipicsasktheexpert@cisco.com going back to July, with the latest one from 29 August. Further, I have the web logs that detail access to my blog posts from the Cisco.com domain, including San Jose and Austin campuses, as well as Cisco remote VPN. Put simply, I got the Motts on this one.

Moving on…

Question 21: The City of Danville, Virginia is one of the early IPICS deployments. Recently the city posted an RFP for “IPICS Maintenance and Operability” that I see as problematic from a security perspective, specifically that there is no mention of patching the IPICS, only one upgrade per year, no detailed requirements for on-call response times (e.g. 15 minutes), etc. — in fact, there is not a single mention of security in the entire RFP. What guidance can Cisco provide to these cities/schools/etc. in terms of managing the IPICS and structuring their RFPs to ensure the security posture of the IPICS is maintained?

Cisco response

Question 22: The IPICS platform is available in two hardware profiles, the IBM MCS (MCS-7845-H1-S31) rack-mounted and the Panasonic Toughbook CF-30 mobile version (IPCM1-P30). Concerning the IPCM1-P30, VMware is used for the IPICS software. As VMware has had several vulnerabilities in 2008 alone, how are these patches applied to the IPCM1-P30? That is, are they included in Cisco-provided updates or must the customer install VMware updates without Cisco pre-testing and supplying these updates?

Cisco response

Question 23: Concerning the IPCM1-P30, what is the underlying operating system that is installed on the laptop that in turn runs the VMware software that runs the IPICS software and how is this base operating system updated and patched?

Cisco response

Question 24: Concerning the IPCM1-P30, what is the operating system running under VMware — that is, is it the same “security enhanced Linux” as the MCS runs, and how is this patched and updated?

Cisco response

Question 25: Concerning the IPCM1-P30, please consider all of the previous questions such as Nessus scans, open ports, etc. applicable to both the base operating system as well as the operating system that VMware is running the IPICS software on.

Cisco response

As with my previous 20 unanswered questions, I look forward to your answers.

Shawn Merdinger
Security Researcher

After my two posts on Tuesday explaining how Aircell was probably blocking VoIP and then why the Phweet/Tringme worked (temporarily), there have been a number of other posts that should be mentioned here:

• Om Malik posted “Aircell: On U.S. Planes, VoIP Will Be Muted” where he relayed a conversation with an Aircell spokesperson that included this classic quote: “we are doing our best to make VoIP services unusable.” Some of the comments on Om’s post are quite good, too.

  Om also relayed that Aircell indicated that Phweet/TringMe will no longer work on their service. As I expected, they blocked the traffic pattern of the service. Aircell will engage in the Whack-A-VoIP-Call and over time they will build up an increasing number of patterns that they will block.

• The folks at Tringme posted “TringMe Conversations (Phweet, Aircell & TringMe Traffic Patterns)” which includes this interesting part (my emphasis added):
  

  TringMe uses TCP and it was a conscious decision. We developed a sophisticated congestion control and packet handling algorithms which allowed us to achieve the advantage of UDP over a reliable TCP connection at good extent. As Dan and others would have noticed, we send traffic in varying small and larger blocks depending on network conditions which is way different from a typical VoIP traffic patterns. This kind of pattern was not meant to break any VoIP blockages, however the goal was to get the best quality even on slower or congested links & we were able to meet the design goal successfully.

  What is interesting here to me is that they do vary the pattern that they use. While this certainly could happen with other VoIP solutions depending upon codec selection, etc., I haven’t seen much variance in actual deployments (outside of Skype, which does vary according to how it is punching through a firewall). The TringMe folks go on to talk about the advantages of TCP (with of course a bit of a sales pitch for why you might want to use their client).

• Dean Foust at Business Week posted “You CAN make VOIP calls on airplanes. Joy.” expressing his curmudgeonly view that VoIP-on-a-plane was not at all desirable. (As I said in P.S. to my 2nd post, I definitely think there are larger societal/cultural issues that we need to work out about whether or not VoIP on a plane is something we really want to have.)

• Irwin Lazar posted on the Enterprise 2.0 blog “American Airlines Aircell Reaction” and made the comment:
  

  Dan York suspects that this will lead to an arms race as Aircell fights VOIP users. I think he’s right, but I don’t think more than a handful of users will care enough to fight the battle.

  To a certain degree, Irwin’s probably right. Some small % of users will actually care enough to try to find ways around Aircell’s blocking. Most folks won’t and will just accept that they can’t use VoIP. Irwin also has some good comments about the impacts of VoIP availability on virtual workers and eliminating the “escape from the office” time that flyers have today. (Some people will want to be “always-on” while others will not.)

• Andy Abramson posted “From the Department of Stupid Is as Stupid Does-Aircell” pointing out that Aircell sells VoIP solutions for airplanes and thus it is a bit strange to see them also saying “No VoIP on planes!” (Although their position is that this is the policy of their customer, American Airlines.)

AIRCELL RESPONDS

Speaking of Aircell’s position, I did receive a nice note from someone with Aircell’s PR firm that stated this:

Thanks for your informative posts about Aircell and the use of VoIP on Gogo today - good reading. We’ve been asked by many outlets for an official response to the VoIP attempts on Gogo so in case you are interested, here is Aircell’s official statement:

It is against American’s policy and Gogo’s terms of service to use VoIP. Aircell has multiple protocols and practices in place to prevent the use of VoIP. Obviously, it is extremely difficult to stop every instance of VoIP but Aircell is monitoring and working constantly to enforce American’s policy and Gogo’s terms of service.

To a certain degree, this statement reminds you that at the end of the day Aircell is simply the service provider implementing the policy of the customer. So it’s really the American Airlines policy that has the VoIP prohibition….. but….

I could go along with that except for one minor little detail. The
Gogo Terms of Service very clearly state the VoIP prohibition. The ToS also states very clearly that the Gogo Inflight Internet Service is provided by Aircell LLC. There is no mention at all in the ToS of American Airlines. Is this Gogo service provided ONLY to American Airlines? Was it created for only American Airlines? Will it not be sold to any other airline?

If it was only created for American Airlines and will only be used by American flights, then sure, the Gogo ToS line up with the customer’s policy. If this is intended to be a generic service and American just happens to be the first customer then I think it’s a bit unfair for Aircell only to be pointing to American. It is, after all, Aircell’s ToS.

But perhaps that’s getting too far down in the semantic weeds…

Technorati Tags:
voip, voip security, travel, air travel, aircell, gogo

After a long hiatus, I’m finally starting to get Blue Box episodes flowing again. I’ve just put up two in the past two days:

• Blue Box #81: iSkoot vulnerability, OFCOM legislation, VoIP security
  news and more (recorded back in May)

• Blue Box #82: Asterisk & Skype security vulnerabilities, new VoIP
  security tools, VoIP steganography, VoIP security news and much, much
  more… (recorded back in June)

And I have some more special editions I’m looking to put out soon. This summer was a bit chaotic for me with a physical move from Vermont to New Hampshire, but I’m hoping things are now settled down enough that I can get back into regular production of these episodes….

Technorati Tags:
bluebox, blue box, danyork, dan york, jonathan zar, voip, voip security, security, podcasts

Update #1: Since publishing this post this morning, I’ve learned of David Berlind’s success using AOL’s video chat from the plane. Odds are that it works for a similar reason to what I outline for Phweet below (and that eventually it will be blocked).

Update #2: As expected, GigaOm is now reporting that Aircell is blocking the Tringme VoIP client used by Phweet.

---

So how did Andy Abramson make a call using Phweet on the new Aircell Gogo Internet service on American Airlines planes? Why was the call not blocked like Skype or Sightspeed calls were?

As I discussed in my last post, VoIP calls have certain network characteristics that make them easy to identify and, in Aircell’s case, block. A VoIP call starts up… a stream of typically 100 packets per second start streaming over UDP between two endpoints… the networking monitoring software notices this, watches it for several seconds, decides it’s VoIP, and blocks the stream.

So why was Phweet’s web-based solution different? Why did it work when Skype, Sightspeed and other services fail?

THE CASE OF THE MISSING PROTOCOL

To understand, we need to look at the traffic from the Flash softphone that is being used by Phweet. The Flash-based softphone is actually from Tringme and is integrated into Phweet’s services. Let’s look at the network capture from a call I had today with Phweet CEO Stuart Henshall using the Flash softphone:

Like the Skype call in my other post, you can see that there is a jump here (at about the 155 second mark) where the Tringme client connects to the Phweet conference bridge. We now have streaming audio, coming in a little less than 100 packets per second (more on that later). Stuart and I talked too long so I can’t easily show you the corresponding dropoff when the call ends (and still use the same scale as my other blog post) but suffice it to say that when the call is over the traffic returns to the previous lower (and bursty) level. Like the Skype calls, the packets sent by the Tringme softphone were small.

So if the pattern is the same, why didn’t Aircell’s sniffers recognize the call and block it? (Have you figured it out yet?)

To see one of my guesses, recall that in my last post I showed how a Skype call was typically all UDP:

Now let’s look at the Phweet call with the exact same UDP display filter turned on in Wireshark:

Oops! Where’s the UDP?

Exactly.

If I look at the traffic and set a display filter to show all the traffic to the IP address of Phweet’s conference bridge, I can see (blue line) that indeed the traffic to/from Phweet is responsible for the uptick in packets:

So if you haven’t already figured it out, let’s bring in a graph showing the other transport protocol, TCP, in green:

And there you go… one very possible reason why Phweet may work and Skype, Sightspeed and others were blocked is simply this:

The Tringme Flash-based softphone is sending audio over TCP and Aircell is not recognizing and blocking VoIP calls over TCP.

Or at least Aircell wasn’t blocking TCP. (They probably are by now or will be soon.)

Now to be fair, if Aircell isn’t blocking TCP, this was probably a decent assumption to make. I mean, the typical mindset to date has been… who in their right mind would send audio streams over TCP?

In all the VoIP systems I’ve worked with, I can’t think offhand of any other systems that send audio over TCP. As part of its range of tricks to get through firewalls, I understand that Skype can use TCP if it is unable to send over UDP, but I’ve never captured it doing so. The IP-PBXs I’m familiar with, both commercial and open source, all send RTP over UDP.

The reason is simple… speed. Setting up a TCP connection involves the “three-way-handshake” and each packet contains information to ensure the packet will get there. If packets aren’t received by the other end, there are requests for retransmission. Data is guaranteed to get there and get there in the right order. It’s perfect for transfers of data where you want to ensure it gets there. However, in poor network conditions, it can be slower due to frequent retransmissions.

UDP on the other hand, makes no guarantees of delivery. Packets are just tossed out on the network pointed at the other endpoint and you hope that they get there. It’s fast, because there’s no involved connection setup, no packet tracking and no retransmissions. If some packets get lost along the way, oh well. Most VoIP systems tend to use UDP because the human ear is okay with a certain degree of lost audio. If some packets get lost along the way, we can probably still make out what the other person is saying. (And cell phones have gone far to degrade our perception of audio quality… but that’s another rant.)

Now in a network with good bandwidth, the performance between UDP and TCP may be similar enough to be okay… but in poor networks, TCP will be slower. For that reason, most folks in the VoIP industry have opted to go with UDP.

So one explanation for why Phweet/Tringme worked on a plane when Skype didn’t is that Aircell made an assumption that all VoIP calls that they want to block would send audio over UDP. The Tringme softphone used TCP instead - and so it got through.

THRESHOLD TOO HIGH?

However, the network trace gives a second possibility for why Phweet worked by Skype didn’t. If you look at the capture from my last post, you can see that the Skype call was coming across right about 100 packets per second:

The math behind this is fairly straightforward. If you are using the 20ms sampling rate common in VoIP systems, that means that there are 50 packets per second (simple way to figure this out - divide 1 second by 0.02 to figure out how many times 20ms is in 1 second). With two audio streams (sending and receiving), the Skype call would generate roughly 100 packets per second.

Now look at the Phweet call using the Tringme Flash softphone:

You can see that the packet rate is not at 100 packets per second and is more somewhere between 60 and 70 packets per second. There’s a simple possible explanation here:

The Tringme Flash client could be using 30ms sampling.

If you increase the size of each audio sample from 20ms to 30ms, you reduce the number of packets generated per second from 50 to… 33. (1 divided by 0.03 to get how many 30ms packets are in 1 second) Double that for the sending and receiving audio streams and you have a total audio packet flow of around 66 packets per second. Which looks awfully close to what is here.

So a second potential reason why Phweet worked and Skype didn’t could be this:

Aircell has set their threshold too high for recognizing all VoIP calls.

It could be that their software is set to trigger on a stream of small packets between two endpoints at 80 packets per second for a period of, say, 5 seconds or more. Anything that hits that rule gets blocked. Maybe it’s 90 packets per second. Maybe it’s 70.

The point is that Aircell may have made the assumption that all VoIP clients would do 20ms sampling and generate a packet flow of ~100 packets per second. The Tringme client used by Phweet does 30ms sampling and so comes in below this threshold.

THE FUTILITY OF FIGHTING THIS ARMS RACE

Now there may yet be another reason why Aircell didn’t detect and block the Tringme softphone used by Phweet. But I would hope that the two potential explanations above would point out the inherent futility in Aircell’s attempts to block VoIP:

There will always be some VoIP client that will find a way around Aircell’s blocks.

If Aircell is blocking on pattern recognition as I’ve discussed in this post and the previous one, someone will make a VoIP client that goes to 40ms sampling over TCP, dropping the packet flow down to a total 50 packets per second. Maybe they’ll use a codec that generates larger packets so that voice traffic becomes indistinguishable from file traffic. Maybe they’ll create a VoIP client that is easily configurable by the person using it so that someone can tweak the parameters while they are on the plane (and subsequently twitter/blog about how they did it).

If they are blocking on protocols like SIP, someone will start using an IAX2 softphone… and when they block that, someone will resurrect an H.323 softphone… and so on.

If they start doing deeper packet inspection, someone will figure out how to use a VPN client that encrypts everything and perhaps fills out all packets so that they are an identical size.

There are a zillion things that people can do to route around and defeat blocking… and in this age of social media, such successes get spread via wildfile (induced by services/sites like Twitter, Techmeme and others). And so many more will learn how to get around Aircell’s blocking… until Aircell improves their blocking… until someone gets around that blocking… and Aircell improves their blocking… and someone gets around that blocking…

Aircell can fight that battle. From their ToS and comments made after this incident, it seems that they will fight this battle and waste their time, energy and resources trying to fight the never-ending cycle of creativity by those who want to go around the system.

Good luck to them.

---

P.S. I do think there are larger cultural issues we need to work out with regard to having access to VoIP in a plane. I do agree with Aircell’s CEO’s comment (in Joanna Stern’s article) that most all travelers have an aversion to people talking loudly on a plane. And yes, access to VoIP calls could definitely make this worse. (I recently tweeted about my experience with cell phones on the train.) But I think we need to figure out some way to deal with that outside of blocking. As Andy said on yesterday’s Squawk Box when we talked about all of this, perhaps there will be new sections on planes where people will be free to talk (or “quiet-zone” sections where they aren’t!).

Technorati Tags:
air travel, aircell, airlines, voip, security, voip security, gogo, american airlines, skype

---

Dan York is Best Practices Chair for the VoIP Security Alliance and writes here and at DisruptiveTelephony. You can follow him on Twitter or identi.ca.

How is Aircell blocking VoIP phone calls from systems like Skype, SightSpeed and Gizmo? (And how did Andy get through with Phweet?)

Ever since last week’s announcement of the “Gogo Inflight Internet Service” provided to American Airlines by a company called Aircell - and the ensuing coverage in the blogosphere - I’ve been getting asked about how exactly Aircell is blocking VoIP calls. Especially after Andy Abramson was able to make a call using Phweet. Aircell is very clear in the Gogo terms of service that no voice calls are allowed:

No Voice Applications. You will not use any type of voice application (including, without limitation, voice over Internet protocol) without written permission from Aircell.

And early users of the system who tried VoIP calls reported that indeed after about 5 seconds or so, their VoIP conversation was terminated. Repeatedly. They could use Skype, for instance, for IM, but not for voice.

So how is Aircell blocking VoIP? (And how did Andy get through with Phweet?)

Unfortunately, I was a wee bit busy last week when all this was breaking, so it’s taken me until now to come up for air enough to write about how Aircell could be blocking VoIP calls on the planes.

THE BIG CAVEAT

First, I should state up front - I have absolutely NO connection to Aircell, Gogo, American Airlines, etc.. I don’t know exactly how they are blocking VoIP calls but am laying out here how they could be blocking VoIP calls.

[I also feel compelled to say that I personally think it's silly of Aircell to block VoIP because there will inevitably be people who figure out ways to route around the blocking. I think Aircell's excuse that they want to block people from talking loudly is also rather lame. I've been on long flights where I was trying to sleep and I've had two people talking very loudly in the seat behind me or next to me. (And yes, sometimes I've asked them to please speak quieter.) Other times I've had babies screaming and crying for most of the flight... or other "energetic" children carrying on. No, I don't really want my neighbor to be in an involved VoIP call but my point is that there are disruptions already. Part of me wonders if there aren't really more technological issues with doing streaming audio/video, but anyway... that's their policy and if you want to use their service, you have to agree. You don't (yet) have a choice in services to use, and you probably won't.]

LET’S PLAY “SPOT THE VOIP PHONE CALL”

Given that Skype conversations in particular are encrypted, I’ve had several people ask me if this means that Aircell can decrypt Skype calls. How else, they ask, can Aircell differentiate between Skype text chat and Skype voice chat? It’s simple really:

Pattern recognition.

A VoIP call in progress has a distinct profile from a network packet point-of-view. In general, the audio streams of VoIP calls (as compared to the call control/signalling channels) have the following characteristics:

1. there are a zillion small packets

2. the packets are sent over UDP versus TCP
3. the packets are sent using the Real-time Transport Protocol (RTP) over UDP

Now with Skype’s encryption, network software can’t know about the contents of the audio stream, so the software can’t know about my #3 here, but the software can recognize the pattern based on the first two. For other non-encrypted services, the software can very simply look for RTP streams.

Now, why do VoIP systems use a zillion small packets? Typically, a VoIP system will sample the speaker’s voice at an interval of every 10, 20 or 30 milliseconds. Most I’m familiar with seem to go for a 20ms rate. So 20ms of audio is captured, encoded digitally and sent off in an IP packet. The exact size of the IP packet will vary depending upon what codec is used to encode the audio. The standard G.711 packets will be at one size and G.729 will be much smaller. Many of the VoIP streams I’ve captured in network traces have a total packet length of somewhere between 35-70 bytes.

To put that in perspective, understand that an Ethernet packet can have typically a max size of 1500 bytes. And packets sent by various protocols can be even larger and will be “fragmented” into smaller pieces (for instance, 1500-byte pieces) to be moved across the network. [Network geeks: Please give me some poetic license here... I realize I could be more precise, but I'm trying not to completely bore the readers.]

The point is that voice packets are typically tiny in comparison to other packets - and there are a lot of them.

How many? Well, if you take a 20ms sampling rate, that means that you are sampling the audio voice 50 times each second… so that’s 50 packets per second for one audio stream. Almost every voice conversation involves two audio streams (one from the caller, one from the recipient) and so you are looking at 100 packets per second for a typical two-way VoIP conversation.

The reason for this is relatively simple. In a file transfer, you are looking to move a file across the network as fast as possible - but you aren’t necessarily in “real-time”. So you generally will stuff the packets with as much info as you can and push them across the network. With voice, we are making use of the fact that the human ear will deal with some lost audio, and so we are chopping up the audio up into a zillion tiny pieces, tossing them in unreliable UDP packets and hoping that enough get there so that the listener can make sense of the conversation.

Put another way, if I wanted to send you this blog post via snail-mail, I could print it out, stick it in an envelope and mail it to you. That’s file transfer. On the other hand, if I were to chop this blog post up and write each word on a post card and stick them in the snail-mail to you, odds are that enough would arrive at the other end that you could assemble them into something like this post. (With luck, maybe you could even make the post shorter!)

That’s VoIP.

So once you know the pattern, it’s fairly easy to spot the calls. For instance, where’s the VoIP call in this network trace?

Do you see it? How about this trace with a filter turned on to show UDP in red?

Ta da… there’s your VoIP call!

Let’s look at this one in a bit more detail:

There it is… all in UDP… and coming in at about 100 packets per second. And if I look at the actual Wireshark traces, I can see that these 100 packets per second are all very tiny sizes. Many of them are between 37 and 50 bytes.

And this is an encrypted Skype call!

No need to decrypt it. Just see that it’s a steady stream of 100 very small packets per second (50 packets per second each way) all over UDP.

Kill the stream. Block it. Conversation dead. No more VoIP on the plane.

It’s basically the network security version of Whack-A-Mole. See a VoIP stream start up… block it. See another one… block it. See yet another… block it. Whenever anything pops up that meets the profile, stomp on it.

This explains, too, why people could talk for a few seconds and then had their conversations terminated. The pattern has to appear in the network monitoring software. The software has to be sure it’s a VoIP stream and not something else… and then the software can block it.

Now I don’t know for a fact that this is how Aircell is blocking VoIP, but it would be easy enough to do it this way.

GETTING MORE SOPHISTICATED

There are, of course, easier ways to kill the conversations. If the VoIP calls use unencrypted audio streams then it’s incredibly trivial to block. Just block the RTP protocol. Period. End-of-story. Now this does involve a hair more packet inspection in that the software has to look farther into the packet headers to see the protocol type, but again this is easy to do. All RTP is typically used for is streaming audio or video… block it and the “problem” goes away.

They could of course go even further into packets and see if they are Session Initiation Protocol (SIP) packets and if so, what the packets are asking to do. If one endpoint sends a SIP INVITE packet to another endpoint and indicates that an audio conversation is to start, the software could again simply block the impending audio stream. (Of course this couldn’t be done if the SIP was encrypted…)

The software could also simply block ports. Block any usage of port 5060 or 5061 and you would probably kill off most “regular” SIP conversations. (Yes, SIP endpoints can make connections on non-standard ports, but the majority of clients probably wouldn’t.) The challenge here is that some SIP endpoints also would use SIP to set up non-audio communication channels like text chat, so blocking all SIP would also block SIP-based text chat which is probably not desirable.

The software could also block on a service level… if it knew, for instance, the host names or IP addresses for the media servers and consumer services (through which the audio would be sent), the software could block all connections to those media servers.

There’s a whole range of additional layers the network monitoring software could use. Any good system will have a “defense in depth” strategy and make use of many of these different algorithms.

Of course, adding on these layers does require more computing power and will undoubtedly add some latency (even on a microscopic level). It may be that for right now they can simply do the pattern recognition approach and shut down VoIP calls.

SO HOW DID ANDY DO IT?

Okay, so how did Andy make a call using Phweet? Given that this post has already gone on this long, I’ll publish my guess in a subsequent post. The text above should give you enough clues, though… any pattern recognition system is inherently fragile because it depends upon recognizing patterns. So what if the audio packets you are sending don’t match any known patterns? What if (hint) the folks at Aircell forgot to watch all protocols?

Stay tuned for some more network charts in my next post…

P.S. This is, by the way, why I think that these type of systems trying to block VoIP calls are inherently doomed… someone will inevitably find a way to “cloak” their VoIP calls so that they are unrecognizable or indistinguishable from other data traffic… it’s a cat and mouse game and inevitably people will find ways to get around the watchers…

Technorati Tags:
voip, skype, aircell, gogo, american airlines, security, voip security

---

Dan York is Best Practices Chair for the VoIP Security Alliance and writes here and at DisruptiveTelephony. You can follow him on Twitter or identi.ca.

Some Dutch researchers recently illustrated once again that “security by obscurity” is not a good way to secure systems. Transport for London (TFL) have for some years been running a prepay card system called the “Oyster Card”. The Oyster Card is an RFID card that you wave over a sensor at the start and end of your train trip, which takes money from your prepay account with TFL. Oyster offers a preferential rate, cheaper than paper tickets, and has had a very high take-up rate with London commuters and residents.

However, the system has now been cracked. At the heart of the Oyster card is a chip called the Mifare Classic, which uses a secret algorithm. Some Dutch researchers decided to target this system, and have discovered how this works. As a demonstration, they used their knowledge to create their own card, which they used to travel around free on the London Underground for a day.

Their interest in the Oyster Card probably stems from the fact that the same Mifare Classic chip is also used in access cards used to secure Government buildings in the Netherlands. In a rather nice demonstration of the separation of powers working in a democratic country, they now have leave from a Dutch judge to publish details of how the Mifare algorithm works, which they will be doing at a security conference in the coming October. This would not have been the outcome that the Dutch government would want, since they now have to take extra steps to secure buildings with more security personnel.

But “hushing the problem up” is not a solution in the security world. The problems don’t go away when you punish the researchers. For every ethical hacker there are probably another two Black-Hats who want to sell the information to those that could profit from free travel in London, or access to Dutch Government property.

“Public scrutiny is how security improves, whether software or airport security or government counter-terrorism measures.” — Bruce Schneier

Welcome back and thanks for tuning in. This is the 4th installment of Cisco IPICS security questions resulting from documentation review and applying my limited security knowledge.

Well, first off, the ipicsasktheexpert@cisco.com is, um, still bouncing (screenshot). Bummer…still some communications are happening on some other channels, so all is not lost.

Moving on…

Question 16: Has the IPICS Server been subjected to the various commercial scanners and fuzzers available? This is not to imply any preference, but scanners like Qualys and CORE Impact come to mind. Clearly, all scanners are not the same — recall the CSA client for Linux port scan vulnerability.

Cisco IPICS Expert answer

Question 17: To what degree are Lawful Intercept features integrated into the IPICS Server? Don’t forget that sometimes folks need to follow what’s going on inside too.

Cisco IPICS Expert answer

Question 18: How well have mature, available security testing methodologies/testcases, such as OWASP and Open Source Security Testing Methodology Manual OSSTMM been integrated in IPICS Server testing?

Cisco IPICS Expert answer

Question 19: Concerning the SNMPv3 implementation, if the IPICS Server operating system is indeed based on RedHat, then it is likely using the Net-SNMP implementation? Even though the implementation is designed for read-only, depending on the MIBs loaded on the IPICS Server, there are potential information disclosures resulting from recent vulnerabilities, such as Technical Cyber Security Alert TA08-162A

On this topic, I highly suggest FX’s “Perception of Vulnerabilities” article.

Cisco IPICS Expert answer

Question 20: The IPICS Server appears to have limited voice recording capability, with other vendors filling this technical niche. In what proactive manner have these 3rd party solutions been technically vetted so as to ensure thet they do not introduce security vulnerabilities into the IPICS Server? And vice-versa?

As with my previously, as yet unanswered 15 questions, I thank you for your time and look forward to your answers.

Shawn Merdinger
Security Researcher

“I don’t want no wait in vain for your love…”–BoB Marley

So here we are with the third installment of security questions for Cisco Systems’ IPICS Expert, questions 11-15. Astute readers (and pesky English majors) will notice that the title is not possessive concerning Cisco Systems’ — tsk, tsk, I had to do this so links in email readers would work and so the title would display correctly in various Web browsers. I suppose I could have gone the Dan York route and added Tiny-URL stuff…perhaps for another post. But, IMHO, Tiny-URL’s are spooky — you never know where they’re going to take you

So, it’s been a couple of weeks now and I’ve still not heard any answers from the IPICS Expert on either of the two previous posts: Asking The Cisco Systems IPICS Expert: Questions 1-5 and Asking The Cisco Systems IPICS Expert: Questions 6-10.

Further, the IPICS Expert email address (ipicsasktheexpert@cisco.com) still bounces…sigh. A little more promising, however, is some nice people from the Naval Postgraduate School I’ve been chatting with forwarded my questions to three or four people in Cisco’s Tactical Operations group last week, so we’ll see what happens. I’m hoping that these new players can move this process forward. If not, at the very least these questions will be out there drifting through the series of tubes on the Interwebs; for whatever that’s worth.

Question 11:

With IPICS Server V1.X, the documentation states that the INFORMIX user password cannot be changed “… do not change the informix password unless you are prompted to do so by the Cisco IPICS installation or upgrade procedure.” This seems to be a limitation, especially in organizations with password change policies (30/60/90 days, etc.). Has this issue been addressed in IPICS Server V2.0, or does changing the INFORMIX user password render the database unusable?

Cisco Answer

Question 12:

Concerning the IPICS ability to limit the re-use of all IPICS Server users’ previous passwords (default is previous 5), documentation on IPICS Server V2.X indicates that this password limitation does not apply to either the IPICSADMIN or IPICS user accounts. To not enforce this policy across all user accounts on the IPICS Server, especially those which are arguably “superuser” accounts, seems quite odd. Please state the justification for this selective application of password re-use limitation that excluded superuser accounts.

Cisco Answer

Question 13:

Cisco Systems has a history of leaving in undocumented hardcoded, “backdoor” and debugging accounts (username/passwords) in several products over the years. Can you please state here, without any question or uncertainty, that all versions of the the IPICS Server do not contain any hardcoded, backdoor or debugging accounts that are undocumented?

Cisco Answer

Question 14:

IPICS Server V2.X documentation indicates that for the IPICSADMIN and IPICS accounts there is neither a maximum number of invalid login attempts threshold, nor a definable lockout period (e.g. 4 hours) after a specified number of invalid login attempts. Considering that these two accounts are “superuser” level and likely targets of remote brute-force login attempts by attackers, is there any active notification such as pop-up alert, email/page alert, etc. to inform the IPICS Server administrator of such bruteforce login attempts? It seems that the only way an IPICS Server administrator would know she is under a bruteforce attack is to manually review attempted logins in the IPICS Server logs; is this correct?

Cisco Answer

Question 15:

During security audits and testing, customers will often use the Nessus Security Scanner to determine the vulnerabilities, if any, of their system. A common issue with Nessus scans are “false positives” during checks. What, if any, false positives can a Nessus scan of the IPICS Server 2.X can a Cisco customer expect? Please include the various scan options (default, polite, sneaky, paranoid, etc.), plugin sets (default, registered, professional, etc.), and especially host-based scans. I suggest providing the nessus.rc files and NBE-format results as well to prove verification.

Cisco Answer

As with my previous ten (as yet unanswered) questions, I thank you and look forward to your answers.

Shawn Merdinger
Security Researcher

Can Asterisk really be used to “unmask”blocked Caller-ID and show the private number?

Well, yes… but it really has less to do with Asterisk then it does with not respecting the signaling sent to you by a SIP trunking provider. It’s conceivable that any IP-PBX could be configured to allow you to do this… and points to a larger issue with trust boundaries between SIP Service Providers (a.k.a. Internet Telephony Service Providers or ITSPs) and their customers.

THE “HACK”

Let’s take a step back first and explain… over the weekend FierceVoIP ran a piece about VoIP security talks at the “Last Hope” conference that referenced a demonstration by Kevin Mitnick of how you could use Asterisk to show Caller ID information for someone calling even if the caller’s ID is set to “private”. Someone (”phant0msignal”) recorded a video of the demonstration (and yes, if you listen, the audio cuts in and out) and posted the video to YouTube and the code to his blog. This might have gone somewhat unnoticed except that it got picked up by Engadget, which naturally garnered a good bit of attention. Here’s the video:

THE EXPLANATION

So was this really a big “hack” that exposed private information?

Not really… although it may be a clever use of scripting within Asterisk.

Here’s the thing:

Asterisk received this information as a natural part of SIP communication because the SIP Service Provider TRUSTED Asterisk to “do the right thing” and NOT display the information.

Which, normally, would be the case. Asterisk would respect the SIP privacy headers and not display the Caller ID. However, in this case Asterisk was modified to NOT respect the privacy headers and display the information that was requested to be private.

To understand this, we need to look at one of the ways that “Caller ID” is usually handled within the world of SIP communication. RFC 3325 defines a SIP header called “P-Asserted-Identity” that is inserted typically by the first SIP proxy that is interacting with the SIP endpoint. The result, within a trusted administrative domain, is the inclusion of one or more headers that look like:

P-Asserted-Identity: "Dan York" <sip:dyork@example.com>
P-Asserted-Identity: tel:+14155551212

The P-Asserted-Identity header, often referred to as P-A-I for short, includes this identity information that can be used by the proxy for the recipient of the call to display “Caller ID” on the recipient’s SIP endpoint (phone, softphone, etc.).

Now, when a call is to be private, there is an additional SIP header included. RFC 3323 defines the “Privacy” SIP header and section 9.3 of RFC 3325 adds an “id” value to the Privacy header. So the resulting SIP headers look like:

P-Asserted-Identity: "Dan York" <sip:dyork@example.com>
P-Asserted-Identity: tel:+14155551212
Privacy: id

Per RFC 3325 Section 7, this Privacy header indicates to the SIP proxy that the P-A-I information MUST be stripped off before the SIP headers are sent to an “untrusted” entity. From the RFC:

Parties who wish to request the removal of P-Asserted-Identity header
fields before they are transmitted to an element that is not trusted
may add the “id” privacy token defined in this document to the
Privacy header field. The Privacy header field is defined in [6].
If this token is present, proxies MUST remove all the P-Asserted-
Identity header fields before forwarding messages to elements that
are not trusted.

So the “hack” in this case was that Asterisk’s SIP handling was modified to NOT respect the Privacy header and instead pass along the P-A-I information to, in this case, the endpoint.

THE LARGER PROBLEM

The larger problem/issue is really this:

Why did the SIP Service Provider send the P-A-I information down to Asterisk box in the first place?

The answer, of course, is simply this:

The SIP Service Provider assumed that it could trust the SIP server with which it was communicating.

The Service Provider extended its “trust boundary” out to encompass the SIP network of its customers. As far as the Service Provider was concerned, the customer was just another SIP network and should be trusted. The Service Provider did not apparently care whether the customer was another carrier - or just someone running Asterisk on a home system. They were simply glad to provide connectivity to the customer.

The problem is:

The trust boundary of the PSTN was then extended out to the customer system.

and there was an implicit assumption that PSTN privacy requests would be respected.

NO EASY ANSWERS

One obvious reaction is “So the Service Provider shouldn’t send that information to the customer’s SIP server!” Perhaps. Perhaps the Service Provider should not trust any of its customers with that information. (And I Am Not A Lawyer so I don’t know if in this case there are actual legal issues here.)

But I’m not sure it’s that simple.

You see, there’s a bit of a “Wild West” going on right now in the world of SIP trunking. Basically, anyone and their brother, mother, father, sister (and…) can get into the world of providing SIP trunks simply by setting up a SIP server (which could be done with Asterisk) and buying some upstream SIP connectivity from a larger SIP Service Provider… ta da… “ZZZZZ VoIP Services” is born. Simple. Easy.

If you are a larger SIP Service Provider, you will sell to smaller Service Providers and naturally extend your “trust boundary” to them. They will sell to others… and so on… and so on… until some final system is connected to some endpoints.

SIP clouds connected to SIP clouds connected to more SIP clouds.

Where do you appropriately define the “trust boundary”? Is it perhaps the “top tier” SIP Service Providers? Is it “the carriers who run the PSTN”? Should it have been stripped off at a gateway coming in from the PSTN?

We’re building this massive “interconnect” of SIP clouds… and this is just one of the many issues that it is not entirely clear that we have a consensus on. Sure, RFC 3325 defines what should happen on a technical level… but what about on a policy level? Who gets to be part of the “trusted” community? (FYI, I would strongly recommend reading RFC 3325 for a better understanding of the issue.)

In the meantime, it’s fairly safe to assume that if you are “blocking” your Caller ID, there is no actual guarantee that it won’t be seen by the recipient. In the vast majority of cases, sure, that privacy will be respected. But there’s no guarantee.

Welcome to new world of VoIP…

P.S. And yes, if you were reading this and thinking “Gee, so can’t the ‘Caller-ID’ be easily spoofed just by modifying the SIP headers?” you are absolutely right. That’s why there’s a good amount of work going on right now in the IETF around the whole area of “strong identity”… but that’s a topic for another blog post some time…

Technorati Tags:
voip, security, voip security, caller id, callerid, telephony, telecommunications, kevin mitnick, SIP, SIP security

“Hello? Is there anybody out there?”

So, it’s been a few business days since I posted “Asking the Cisco Systems’ IPICS Expert: Questions 1-5″ and while I haven’t heard anything back from the IPICS Expert either via email or comment on the blog post, it is somewhat amusing, and perhaps a bit disturbing, that a Google search for “IPICS Expert” leads back to VOIPSA…go figure.

Anyway, as with the previous post, this post continues focusing on Cisco Systems’ IPICS (IP Interoperability and Collaboration System) Server, the “heart” of the IPICS solution, with five more questions for the Cisco IPICS Expert:

Question 6: Early versions of the IPICS Server documentation refer to the operating system as Red Hat Linux, while a later version of documentation refer to the operating system as “Cisco Linux” and the latest version of documentation states “Linux” — Is the IPICS Server still based on Red Hat? If so, what version of Red Hat (enterprise, etc.)?

Cisco Answer

Question 7: Does the IPICS Server have any kind of file-integrity assurance program like, for example, Open Source Tripwire?

Cisco Answer

Question 8: Is the “Cisco Security Agent” provided at no cost for the IPICS Server, or is there an extra cost for this piece of software “protection?”

Cisco Answer

Question 9: The IPICS Server uses the IBM Informix database. According to documentation, IPICS Server 2.1(1) uses IBM Informix Dynamic Server Version 10.00.UC1. In 2008 several vulnerabilities were released concerning Informix, such as CVE-2008-0949, CVE-2008-0727, CVE-2008-0768 , CVE-2008-0369 , and CVE-2008-0368. If applicable to the IPICS Server 2.1(1) and earlier versions, have these vulnerabilities been addressed and patched in the IPICS Server? There seems to be nothing at the Cisco PSIRT site addressing these vulnerabilities. Am I missing something here?

Cisco Answer

Question 10: For IPICS Server 2.1(1), please provide a listing of all installed RPM packages, their version, and indication of known vulnerabilities in each RPM package.

Cisco Answer

As with my previous five (as yet unanswered) questions, I thank you and look forward to your answers.

Shawn Merdinger
Security Researcher