Understanding MPLS VPNs

If you are an enterprise user of carrier WAN offerings, it is likely you've been offered MPLS as a solution. Most carriers are encouraging customers to consider MPLS based services over traditional Frame Relay, ATM and even Point-to-Point transport.

One misconception is that consumers of MPLS carrier services must "run MPLS" within their own networks or at least on the edge device(s) connected to the carrier's MPLS service. This is not the case (unless you are doing something pretty unusual). Standard routers -- and even bridges -- are used on the customer-side. The configurations may be a bit different than you're used to but they're still relatively straightforward (and cookie cutter once you do one).

Your layer 1 and layer 2 skills will still come in handy. The underlying transport is still going to be TDM (DS1, OC-3c, etc). You may end up running a dynamic routing protocol (BGP, OSPF) with the carrier's network. If that's new territory, don't worry. This BGP configuration is far less elaborate than that needed to (prudently) bring up BGP for Internet multi-homing.

So even though the MPLS component will be outsourced to your carrier, to be an informed buyer and troubleshooter when shit-hits-the-fan, you'll want to understand the different ways that MPLS can be delivered and used by carriers to provide your service. When the carrier asks you to make some choices or you're evaluating a prospective solution, you're more likely to get what you need (and hopefully less of what you don't).

Jeff Doyle, the author of Routing TCP/IP Volume I and II (both 900+ pages each), has two quick articles about MPLS. In Part I, he covers the basics relevant to any MPLS user as to the different types of MPLS network options and in Part II he covers some of the nitty gritty relevant to service providers and those with an interest in what goes on behind the scenes.

P.S. I have some experience with WANs. Feel free to ask me questions. You can post a comment here or drop me an e-mail.


-jr

Is Risk Aversion Our Greatest, Uhm, Risk?

Are WE Holding Ourselves Back?

(This is a draft of an informal essay I wrote today. Figured this would be a good place to post it and garner some feedback). A brief excerpt:

"Often we're concerned about failure. The great irony about the perception most humans have about taking risks and failing is that it nearly insures that most of us will, in fact, have the greatest failure of all: never seeing our most fruitful ideas turned into reality and achieving our most important goals in life. Our built in risk aversion is really, quite ironically, our greatest risk of all. A wolf in sheep's clothing."

"The truth is that most of us can handle far more “risk” than we currently do. On the other hand, we could do with a lot less of the risks we do chose to take on...."

Most of us are capable of far more than we give ourselves credit for. Fortunately, we are the only ones holding ourselves back.

Usually due to a combination of starting something worthwhile but not finishing it and coming up with a good idea but not doing anything about it (taking action), we stand still or, at best, make very very slow progress. Thus, at best, even if we make some progress towards our goals we still do not end up actually passing the goal line.

Often we're concerned about failure. The great irony about the perception most humans have about taking risks and failing is that it nearly insures that most of us will, in fact, have the greatest failure of all: never seeing our most fruitful ideas turned into reality and achieving our most important goals in life. Our built in risk aversion is really, quite ironically, our greatest risk of all. A wolf in sheep's clothing.

Our perception of the risks of most failures are outlandish. While there are certainly some things that are risky enough they could, say, kill us outright, most failures are far less dramatic. Some types of failures can be quite stressful to be sure. Some may even shorten our lives by a few years (due to the stress, though even a bit of short-term stress can sometimes be worth it if it makes the remaining years that much more satisfying). But nowhere near the percentage we think -- of “risky ideas” that we all come up with in our day to day lives -- are even half as horrifying in impact, if we were to take action and fail, as we might convince ourselves they are.

Our comfort zones hold us back. However nearly all good things that come to us, arise from somewhere outside of our comfort zones. Taking on our first real job. Driving for the first time. Taking an entrance or certification exam for a college program, to teach, or some other program we want to pursue to push our careers forward. Marrying for love. Having our first child. Flying for the first time. Learning to swim. Passing a difficult test that forced us to really learn the subject matter rather than simply memorizing a few key concepts. Learning to take our first step (though most of us will lack firsthand memory on this one). Asking someone attractive (in whatever way you deem important) out on a date or even simply for coffee. Starting a blog and posting our real thoughts, opinions, and ideas out there for the world to yell back that we're wrong. :)

While each of these can be stressful in the moment, that feeling soon subsides (especially with practice and time). Without these stretches, life would be so boring and, well, lifeless. We grow, becoming more comfortable in our new terrain. When viewed with a receptive mind, we even learn a lot from our failures.

Nearly all “firsts” in our lives are outside of our comfort zone. In fact, some of them may even be far more realistically life threatening than the other ideas and opportunities that we chose not to take action on. So much for our built-in perceived versus real risks radars.

When I was starting my most recent consulting business I knew there was a good chance that cash would get a little tight for a while. Since I knew that was a high probability outcome along the way towards my goals, I could plan to address it. I could take some actions to handle the looming issue and I could think through some of the options I'd have, depending on how bad things got when the time came. To me, that wasn't really a risk. I trusted myself and thought my way through it. There are few situations in life where we have absolutely no options. It wasn't that I didn't worry about having money to pay the rent and buy food. It wasn't that it didn't stress me out. It was more that the real risks that scared me more than the others were the things that I might have fail(ed) to anticipate and plan. To a certain extent, the ones entirely (or mostly) outside of my control, were a big deal but, again, it's all about having options. As long as I was confident I'd have options, the number of real risks in my world quickly shrunk and became manageable.

The truth is that most of us can handle far more “risk” than we currently do. On the other hand, we could do with a lot less of the risks we do chose to take on....

Our perceptions that result in us not taking on risks that we should while continuing to do things that we shouldn't are even more humorous when considered in another light. I got my first credit card when I was eighteen. It was an American Express. A Mastercard soon followed. At first, I had the money so it really wasn't a big deal. Then I left my comfy job to try my hand as a pseudo-partner in a friend's business venture. That fizzled out. I had some savings from a well timed stock option sell-out. It didn't take long to burn through that. After all, I'd gotten used to a pretty good salary (even if I hadn't been only eighteen at the time). I temporarily struck out on my own (consulting without any specific plan other than to explore new business opportunities) and then, a short time later, became a partner in another new business venture. Well, my financial situation changed quite a bit over that time period. And, like many early entrepreneurs without a solid win under their belts, my partners and I didn't pay ourselves much since we were in start-up mode. But, hey, I didn't have to change my lifestyle – I still had all those credit cards, right?

Give nearly anyone a few dollars and they'll have no problem finding a way to spend it to get something they need (let's not worry about the distinction between need and want for today). Now combine that with easy access to credit (credit cards and home equity loans are the most common current incarnations). Coupled with the basic desires that we all have to please ourselves, get a bit of instant gratification from time to time, and reward ourselves for a job well done or some ill we suffered that day, and our perceptions of risk go out the door.

Suddenly we're no longer thinking about how we'll afford to pay off that large credit card balance next month, how much extra we'll really have paid for today's little indiscretion due to the compounding interest we'll have paid before the balance is gone months or years down the road, and, worse in my mind, the opportunity cost that slowly at first and incrementally over time builds up until we have convinced ourselves that we “just don't have the money to do whatever we want”.

We want everything now so much that we put ourselves in a permanent position of never actually getting what we want. Irony can hurt, especially when it's wired into the standard operating procedure of our brains. It's a bit like the inverse of “wanting to have our cake and eat it too”. We use perceived risks as excuses not to do the things we really should if we actually want to achieve our goals. And we toss out the real matter-of-fact risks when it comes to acquiring the things we could probably do without for just a bit longer. If only... If only...

“I want it now, the future be damned!” Don't get me wrong. There's a time and a place for this attitude – it can be what gets us through some days. We're all human and I doubt we're supposed to be perfect all the time. Besides it's no fun to be perfect. The problem is recognizing when it has become a habit, a regular occurrence, and something that we keep doing even while making excuses about not doing the things we know we really should. (Sadly it can become a feedback loop unto itself, it almost being worse if we are aware that this is what is going on but don't have the strength left to pull oneself out of it so we feed the indiscretion monster more to get through each day and it gets worse -- so watch out!)

While we can be our own worst enemies, remember that is a good thing as well. It means it's under our control. While it's not easy to fight what is hard wired into our own brains, it can be easier than many other battles we participate in outside of ourselves. It's certainly a more important (and probably much more effective) fight. I challenge you:

1. What is one really attractive goal you have?
   
2. What step, or even steps if you are really on it, have you taken in the last day to get you there?
   
3. What about in the last week?
   
4. The last month?
   
5. The last year?
   
6. The last decade?
   
7. Don't beat yourself up over the answers to #2-#6. More importantly, what are you going to do TODAY?
   
8. Now, to make it a little easier to stay on the ball tomorrow with your new ambition, what is something you can do tomorrow as well?
   
9. And the next day?
   
10. And the next?
    
11. Good work --- keep it up! Momentum has a tendency of building, even from nearly nothing. You'll be there in seemingly no time if you keep it up. But you do have to START somewhere. Get moving. NOW.
    

-jr

VOIP Troubleshooting With (the free) Wireshark Packet Analyzer

Wireshark is a network protocol analyzer. Some may recognize it by its former name, Ethereal. It's free (and open source), runs on multiple platforms (including Windows and Linux), and actively developed. For those doing VOIP installations or troubleshooting existing installations, the latest release has some very handy VOIP specific support.

It will create visuals representing captured SIP and associated RTP connections. You can drill down by clicking on specific spots on the graph to pull up the associated packet(s). You can generate reports (as well as graph) jitter, bandwidth usage, etc. Various ways of displaying the data to get a better idea of what's really going on.

The screen captures at the beginning of this post are from Wireshark. They show a graph of a VOIP (SIP) call (and a half) between two Snom SIP phones attached to an Asterisk-based PBX (the green/blue/purple image). And an analysis of the associated RTP session (including packet loss, jitter, delay). WS can even playback captured VOIP calls (at least if using PCM/G.711/ulaw).

-jr

PCI: It's Alright to Question the Auditors

"It's alright to question your PCI auditor. This isn't about getting out of doing things that really should be done. It's about making sure you aren't unnecessarily wasting money, period. Ask them to justify their findings and recommendations. And seek a second opinion (from another auditor or a security expert) if need be."

A bit back a (then prospective) client came to me while going through a PCI audit. They'd been informed by their auditor (VeriSign in this case) that they needed to segregate off a group of servers. Fair enough. The catch was they were also being told that this needed to be done using a second firewall, in order to be compliant, even though their existing firewall had more than enough interfaces to configure additional distinct security zones. The proposed second firewall would be under the same administrative control and offer no greater granularity in security policy enforcement. In short, it wasn't a terrible idea but it didn't seem very value enhancing either.

The client had an inkling that this shouldn't be necessary. They had further discussions with the auditor to no avail. In the interest of time and manpower, they went ahead and bought another firewall. I was called in later to integrate this and some other changes into their network. One of my first questions was "Why are we doing this?". After hearing a bit more of the background I still felt firm in my conviction that either (a) we weren't getting the entire story and thus even with a second firewall I wasn't sure we were meeting the requirements or (b) there really wasn't sufficient grounds to add a second firewall when the isolation could be done completely adequately on their existing firewall by shifting around the topology a bit to utilize available interfaces and adding some new access rules.

My view was that the assessor had a specific ideal model in mind and wasn't really listening to the arguments given thus far. This was even though those arguments weren't against the server isolation being suggested. The only disagreement was over how to get the end result.

In the interest of time I proceeded with preliminary integration plan development that included the second firewall while recommending a continued push that the auditor needed to justify their recommendation more specifically. Over the course of the next several days, after the client had gotten input from myself including points to bring up and gained additional confidence in their original inkling that the extra firewall was unnecessary, the auditor shifted gears and said implementing the requested isolation on a single firewall was acceptable.

At this point I'd only spent several hours on this project. There was no longer justification for the purchase of a second firewall and the changes required to isolate the servers were far simpler. Even though my client had already purchased the second firewall prior to my involvement, they could now return it, sell it off, re-deploy it elsewhere, use it as a spare, etc.. The expense of engineering and labor for a more complex integration effort was avoided (plus, the long-term costs of having another piece of equipment to maintain, an added failure point, and a more complex topology to troubleshoot).

There are something like one hundred or so assessors that work with the PCI Council to do audits. Each has their own strengths, weaknesses, and agendas. Some are relatively pure-play professional services providers while others sell their own security software and hardware (and, yes, often related to assisting you in gaining PCI compliance). Assessors are allowed to recommend their own services and products as solutions to problems that come up during audits (though they are not supposed to require their use in order to pass). The PCI DSS standard isn't specific -- which is actually a good thing since every environment is different -- so there's much open to interpretation at both the end-user and the auditor level. Finally, all auditors are human and make mistakes as well.

Bottom line: it's alright to question your PCI auditor. This isn't about getting out of doing things that really should be done. It's about making sure you aren't unnecessarily wasting money, period. Ask them to justify their findings and recommendations. And seek a second opinion (from another auditor or a security expert) if need be.

-jr