Friday, August 24, 2007

Getting a Network Lab on the Cheap

A network lab is useful for all sorts of things. Unfortunately really good labs can be expensive to put together. Over the years I've developed some alternatives that have worked well for many situations. This is still in informal draft format but I don't know when I'll next get a chance to clean it up. I think some folks may find it useful as is so I'm going ahead and posting it now.

Here are my strategies:

  • Emulators
    • There are Cisco hardware emulators that allow you to run IOS and PIX/ASA images. This has also been known to be possible with other vendors from time to time, sometimes officially offered by the vendor and sometimes not. I suggest a Google search for something like "vendor emulator".
      • Dynamips (Cisco hardware emulator that runs your provided IOS images for the 3800, 7200 and other platforms).
      • PEMU (Cisco 525 PIX hardware emulator that runs your provided PIX OS image)
      • Versions for Linux and Windows of the above
    • These emulators often run in their own VMs so, for example, it's possible to set up an entire lab of Cisco devices on a single laptop or desktop and have the emulated devices all "connected" to each other purely in software. e.g.
      • 2 x Cisco PIX 525 + 2 x 7200 NPE-400
      • ....on a 1.4Ghz Celeron laptop
    • Don't expect to do performance testing with emulators but they are often fully functional (the Cisco emulators run actual IOS and PIX images, whatever ones you provide) since you can build testbeds that closely resemble your actual network, including testing actual config changes, etc.
  • Vendor and reseller online accessible demo environments
    • These are nice for poking around and following along as you read through their manuals. Sometimes you can't change things but you can usually get as far as saving your changes -- which is good enough for many situations.
    • Can also be useful to see what newer versions of firmware look like, before you upgrade, since vendors usually keep their demo units up to date
    • Do Google searches for "myvendor demo", "myvendor demonstration", etc. e.g.
  • Quasi-Public looking glasses
    • Telnet reachable looking glasses / route servers / traceroute servers
    • Web-based
    • usually Cisco, Juniper, and some Zebra-based Linux/BSD boxes
    • Not as useful as other methods but sometimes can view live info you might not be able to simulate elsewhere such as large BGP tables
  • Rack rentals -- rented remote network lab racks (most often rented in 4-6 hour chunks, designed for folks studying for certifications such as CCIE)
    • as cheap as $10 for a 4 hour block (!!!)
    • check eBay for low cost rentals
    • check Google for others -- there are many so prices are pretty competitive
    • search for "CCIE rental" for Cisco rentals. Similar ones exist for Juniper and other vendors with certification programs.
    • since folks studying for higher end certifications, such as the CCIE, require pretty elaborate labs these "rack rentals" give you access to some incredibly large and higher-end equipment
  • Quasi-public environments (left open by accident? Sometimes folks leave their mgmt interfaces open intentionally, accidentally, or, hopefully, just the read-only modes)
    • When evaluating Axis cameras a while back, I did some Google searches and turned up a bunch of cameras left open to the public.
    • Hint: For web-based GUIs, figure out a unique portion of the text string that is usually in the HTML title and search for it using Google's "intitle:" command.
  • Try and Buy promotions
    • Many vendors, especially the smaller and more aggressive ones, are pretty open about letting folks try their products out for a month or so. Many times you can even talk 'em into holding onto them a little longer if you keep in touch with them, let them know you are evaluating for future purchase, etc.
    • Helps to have a business entity, be a "consultant" (state outright that you regularly recommend equipment to clients, if that's the case, since that peaks the interest of most sales folks). Even better is a tax reseller ID I suppose (I don't resell hardware so haven't taken advantage of this last bit before but I've been asked by sales folks when evaluating hardware who I imagine like to encourage new potential sales channels).
  • Purchasing standby spares
    • Spares bought to have for on-site swap-ins in the event of hardware failures can be used (carefully!) for lab work. Just be careful not to break, lose, or otherwise leave your spares in a state that hurts their real function.
  • Purchasing, borrowing, decommissioning lower-end but current enough models
    • Older models, that may or may not be end of sales status by the vendor, may be available cheaply. It's all about eBay.
    • Upgrading production devices and retaining the older models that get pulled for the lab (previously sunk cost, easier to justify over spending more $$$ on lab equipment)
    • A maintenance window for a non-critical office/PoP or during a time window (e.g., after 5pm, or 2am-6am or whatever) that is acceptable in your environment. Then temporarily doing the testing/evaluation/learning with this equipment before returning it back to production state. This isn't ideal but works better than "learning" on a more important part of your network at the same time you are intending to go live with the changes.
  • Out of date models that people are giving away
    • for some needs, "free" equipment can meet certain needs, especially when first learning about a new device when just getting the basic look and feel of the command line or GUI is the focus. Spend money only when you get a bit farther along -- after all many times projects get delayed mid-stream anyhow so why buy new lab equipment only to have it sit and collect dust for the next few months until you get around to working with it (and probably frustrating your boss who you convinced to spend money on it now rather than later).
  • Borrow from a colleague
    • Call your colleagues at other organizations. Often they've got some extra hardware sitting around that they are willing to lend.
    • If you break anything make sure you tell them upfront and replace it (been there done that, with an awful hardware problem in a modular router that, best as we could tell, took out one of the test chassis in addition to the problematic one due to some weird short on a removable module we pulled from the busted chassis to test in the known good chassis).
  • Borrowing under vendor hardware replacement and support contracts
    • If you have a current hardware replacement or support contract with your vendor, you can explain your particular situation and sometimes have them do you the favor of lending you equipment temporarily.
    • In a near-emergency, if you have an advanced hardware replacement contract, you can consider having a "failure" then sending the advanced sent hardware back when you're done with it. Don't abuse too much Might be acceptable under some circumstances. Make your own judgment.
  • Hosted x86 Virtual Machines
    • usually VMWare or Xen based (not that it matters much to the end-user)
    • can boot up various OSes on-demand, cheaply
    • some hosting companies will let you rent on short-term basis (e.g. Amazon EC2 Xen-based VMs charge by the hour and don't charge you while your VM is "shutdown/turned off")
    • Good for adding hosts to your test lab (nobody ever said a useful lab had to be physical)
    • Combined with network device emulators, discussed previously, makes it possible to do some crazy stuff
      • Want to simulate a 50 router nationwide network of Cisco 7200 class routers? Get a usage-based VM hosting account and install Dynamips.... It'll cost you something like $20 to get access to 50 VMs for four running hours via Amazon EC2
    • If you've got some real equipment you want combined with your virtualized equipment, remotely hosted x86 VMs, etc than build a VPN overlay to connect things up the way you want. The VPN can be an invisible transport layer for your lab network if you want or it can be part of the lab network itself.

Wednesday, August 22, 2007

Routers, Switches, and Firewalls: Marketing Benchmark Numbers versus Reality

A consistent problem when reading datasheets for networking devices (routers, switches, firewalls) is that the throughput numbers offered by the vendor are not useful without context and are coming from a bias source. Today I happen to be reading through a SonicWALL firewall datasheet and I notice a reference to RFC 2544 in the fine print:

**Testing Methodologies: Maximum performance based on RFC 2544 (for firewall). Actual performance may vary depending on network conditions and activated services
***VPN throughput measured using UDP traffic at 1280 byte packet size adhering to RFC 2544
****Throughput measured using HTTP throughput test
Well, RFC 2544 has apparently been around since 1999. It suggests a framework for a standardized methodology for benchmarking networking devices. I'm not sure why I've never come across it before. (Note: I haven't read through it yet so I'm not endorsing it).

Thursday, August 16, 2007

I Don't Know. Really.

Sometimes we don't really know the answer but pretend we do. In fact, sometimes may be an understatement. Not knowing the answers -- or having enough data to have an informed opinion -- but pretending we do is not the foundation upon which to have a discussion to help yourself or someone else arrive at a more informed opinion.

This is particularly important with complex worldly issues (say, geopolitical problems that have the potential to create wars). Few of these types of issues have truly black and white answers. The "truth", such as it is in these cases, often lies within carefully selected -- yet still meaningful -- nuances that can only be honed after significant study and analysis.

Best case, we come off silly. Worst case, we, well, kill a few people. Thankfully we're all adaptable and like to better ourselves. So we can get better at all of this.

Get out there and vote but become truly informed first -- don't just sound informed to those that already agree with you. Be able to have an honest opinionated discussion with folks that don't agree with you and still walk away with an understanding of where they are coming from. If you can't do that, you probably don't know what you're talking about -- and should get back to reading, researching, and thinking before opening your mouth.

Anyhow, the excerpt (along with watching lots of West Wing episodes) that inspired this post (even though it was talking about managing software projects) is below:

True Factors

Next time someone tries to pin you down for an exact answer to an unknowable question — whether it's for a deadline date, a final project cost, or the volume of milk that would fit in the Grand Canyon — just start by taking the air out of the room: say "I don't know."

Far from damaging your credibility, this demonstrates the care you bring to your decision-making. You're not going to just say words to sound smart. It also levels the playing field by reframing the question as a collaborative conversation. By learning how exact your estimate needs to be (and why), you can work together to develop a shared understanding about the true factors behind the numbers.

—Merlin Mann, creator and editor of

Tuesday, August 14, 2007

More To Lose, Priceless Lessons

....and Why Wealth Is Best Acquired Neither Easily nor Difficultly
....and Personal Financial Literacy

I manage my own investment portfolio. For the most part, other than experimentation for educational purposes, I keep things pretty simple. I don't short stocks. I ignore options. My interest in the performance of the major indexes (e.g. Dow Jones Industrial, NASDAQ Composite, S&P 500) is mostly for entertainment purposes. I rarely consider fixed income securities (e.g. bonds, treasuries, etc), except during unusual situations when, effectively, due to the thesis of the investment in question, it really isn't a fixed income investment I'm making. I take long-only, and usually long-term oriented, positions in publicly traded stocks. Fractional interests in real businesses. Plain and simple.

My present approach, and seemingly most likely to remain permanent (though it will continue to go through maturity spurts as all good conclusions should), is based on quantitative fundamentals (i.e. the numbers on the balance sheets, cash flow, and income statements) combined with deep qualitative analysis. Purchase and sale price determination is based on a value-driven approach somewhat akin to Benjamin Graham, Warren Buffett, Third Avenue, Longleaf, Sequoia, Mohnish Pabrai, and others. The math is not rocket science. In fact, if it wasn't for the surrounding analysis, my eight year old could probably be taught how to do it. If I have to whip out spreadsheets and make lots of assumptions it's probably not the investment for me.

Call me old fashioned, but I invest to make money and that also means doing so as efficiently and effectively as possible. I see no reason to look at higher branches when there are inevitably enough low hanging fruit around (which also happen to be where I'm less likely to break my neck if I slip and fall). I enjoy investing but I also have plenty of other things I enjoy spending my time on.

With varying degrees of success I've been doing this for the last decade. Unfortunately it wasn't until the last two to three years that I finally really had enough of the pieces together to be able to deliberately and confidently make transactions in which I could be more certain that the outcome was due to my careful analysis over simple dumb luck. So my lessons have been both highly profitable and, alas, combined with others, highly expensive. i.e. I'm not (economically) wealthy yet. :-)

I wouldn't take any of it back though for anything. Not even a million dollars. Why? Glad you asked. :-) Two main reasons, more to lose and priceless lessons, which are sort of just opposite sides of the same coin. You see if I'd started with a million dollars the lessons would have been proportionally that much more expensive. Investing is a cumulative learning activity where you can learn a lot from your mistakes and where constant reading, honest reflection, and careful analysis is necessary. It's also useful to have a long-term perspective and understand that short-term performance is not something that can actually be extrapolated from in estimating long-term performance. (Side note: Rarely do things go quite as planned, especially so in the short-term. In the longer-term, things are usually closer to plan. So, uh, plan accordingly?).

If I had started with a million bucks I'd not only be much poorer than I once was (relatively speaking) but I'd also have spent even more on those priceless lessons I've learned along the way. Worse, I might have been just too risk-averse to even learn some of the lessons I have -- after all there would have been more to lose. The lessons may have been priceless but it's still much easier to earn a few thousand (or tens of thousands) back than a million or two when you have to start all over from zero again.

Another thing I like about this path is that extreme wealth, without having great and painful lessons along the way to achieving it, can often make one too conservative and controlling and naive for their own long-term economic good. For example, having all your capital tied up in fixed income securities forever won't do much for increasing your buying power. (Or, I suppose, not being controlling enough is also a problem if you are a spendthrift partier which is really the exact same problem manifest in a different form).

Even if one has painful -- but useful -- lessons one may still not reflect upon them properly and/or for some reason or another not behave economically rationally when the rubber actually meets the road. Unfortunately the same is true for most of us who have never truly achieved significant wealth. We usually have incorrect or irrational attitudes amount money and wealth, managing it, saving it, growing it, etc. We are just as bad as many of the folks that inherit their money without good parental financial teachings or, at least, innate (rare) financial intuition. We just aren't quite as first hand knowledgeable that we're pissing all our wealth away. i.e. We don't have the historical memories of seeing all the zeroes in our bank account balances and thus we are even less convinced we even have the potential to have larger bank accounts ever -- lot alone again. We're more ignorant of our potential because we didn't have it once then lose it (yet). :-)

While you can't deposit potential in the bank, having it and not capitalizing on it (oh, a pun, heh) is just as awful as already having lots of wealth and pissing it all away. And, if you really stop to think about it, it probably hurts just as bad too.

I suppose the conclusion is that becoming wealthy should not be too hard nor too easy and it's important to pay attention to the hard learned lessons along the way without getting too down on oneself. At least if your goal is to achieve wealth and "keep" it.

Unfortunately, probably much to the chagrin of economists, we humans are not economically rational beasts as a rule. We're so full of exceptions, biases, heuristics, assumptions, etc, at least if my own experience is of any commonality, that I couldn't even begin to comprehend what rules we all do operate by. Collectively we're fairly economically rational but individually we're not so much. In some ways, that explains why wealth, at least great wealth, tends to collect in fewer hands rather than being broadly dispersed. This seems true not only within wealthy societies but geopolitically as well as best as I can tell.

So, anyhow, this brings me to the one investment, wealth, and personal finance related topic that I have been struggling with. You see, I get freaked out a lot these days when I talk to friends, family, and various other folks about their investment portfolios. Not because I think I'm super smart investor guy whose feet everyone else should bow to. Feel free, however. j/k :-). I just happen to have already learned the hard way (and, because I discovered a deep interest, and tried to take it up a notch as well, I'm ultra-aware of this problem domain).

Let's get more specific. It's not uncommon to hear that someone has bought some stock for their portfolio and their reasoning -- and it's usually the extant of it -- is that it's a "cool company", "someone I know told me about them and I trust their judgement", "growing lots", "in the news all the time", "safe bet that can't lose", etc. etc. Yeah, that's great but what's the investment thesis? How can one be sure the price paid was realistic? How will one know when it has reached or exceed fair value (at what price shall it be sold)?

Oh, yeah, that. Buying a security without an idea of what it's worth is like saying you'll consider yourself happy when you're successful; We can all understand the words in this proclamation, their individual meaning is straightforward enough, but the sentence is too vague to be beneficially to anyone including the person stating it. Define success my friend before you attempt to achieve it. Keyword being before.

Just because, and even if you are "right", the stock of a company you bought goes up in price after you buy it doesn't mean it'll go to any particular price. Buying stocks is not as simple as saying "I'll just buy and sell when it goes up". Goes up to what exactly? Oh, double what you paid for it? Well, any idea if the company is even remotely, even under the most optimistic valuation, worth that much? One in this circular situation may be relying more on hope more than prudence. Worse, they are guaranteed to not be able to remove emotion from what should be an entirely business-like transaction. It's hard to remove emotion from the transaction when the initial purchase reasoning was hardly a step beyond that.

Another metaphor at the risk of beating a dead horse: Committing to a project, without confirming the project is even worthwhile, and stating at the same time that we'll consider it complete when it's done. We'd probably define "done" somewhere. Perhaps we'd use a metric of some type or at least some outcome that is a bit more readily identifiable and tangible. Oh, and we'd probably not want to spend our time on the project if it's not worthwhile (e.g. overpriced or just not the optimal return for the risk or, worse case, pure hype) so we'd make sure to evaluate that before we jump in too.

It's not that any of these people are stupid. Mostly they are in different businesses of their own, they are busy enough with enjoying their lives, they just have no time regardless of interest level in understanding this stuff, they don't know any better and take too many clueless folks or conventional wisdom articles at face value, they have zero interest in understanding this stuff (nothing wrong with that), they think hiring outside assistance is too expensive, etc. None of them is stupid at all. They just have other things going on.

So a goal I have is to figure out how to help improve the overall personal financial literacy of the populace. After all, we're all in this together and zero-sum game or not (depending on your view), there's plenty to go around for us all to be more than well off.


Friday, August 10, 2007

SCO Loses Linux Battle and Personal Investing Update

Linux is Legal (Still)!

Today SCO (the idiotic falling over themselves almost to the point that I almost felt sorry for them but decided to simply laugh at 'em instead company that has claimed broad intellectual property rights over the Linux kernel and has lawsuits pending against Novell, Red Hat, and IBM) received a big blow today in the three year old court battle with Novell over whether or not SCO even owns and controls the IP they claim to (they bought/licensed/borrowed? so-called original UNIX from Novell in 1995).

Short answer: Nope, nada. Read more elsewhere on the web or take a look at which has followed the whole thing since the beginning. Sorry, Darl, but your claims have never made any sense. On the upside, SCO is now a household name (again) within the Linux and IT communities. Too bad, its got sh*t all over it too now. Maybe Darl is really a super secret undercover marketing researcher mad scientist who was just out to prove wrong the clique that "there is no such thing as bad publicity".

Investing and Temptations

A couple months ago, in order to insure I would not make any foolish decisions and to stay focused on near-term cash flow sources (such as my consulting business), I stuck most of our uninvested cash into government securities. You see, a serious hobby has become analyzing public companies and investing in them when a (rare) gem at a good (or, more rarely, great) price pops up on my radar. The plan was (is) to leave things alone for six months or so. It had nothing to do with believing it was the best place for our cash -- I just didn't want to have too many distractions keeping me from focusing on what should be my near-term priorities. And, knowing I had other things to be worrying about, I didn't want to risk half-assing my investment analyses.

Well, then the market had to spin a bit. Most folks are looking at their holdings being down the last few weeks. A few of the stocks on my On Deck and Watch lists have dropped into more attractive price ranges. I really shouldn't be spending time on this right now, I have a consulting business to build, but damn it -- some of these look to be pretty attractive transactions to step into now and let play out over the next one to five years. Doh!

(And, no, SCO's most definitely not on the candidate list).

Thinking Differently About Problem Solving

We are obsessed with coming up with solutions but rarely do we step back to truly consider the most effective process for generating optimal solutions consistently. And we're quite reliant on mental heuristics, which are certainly helpful in our day to day lives, that deceive us into making intuitive but sub-optimal decisions in ways we are unaware of. And, finally, we're influenced by conventional wisdom which may not be so, well, wise.

A nifty (only 19-page) essay on the topic of generating optimal solutions more consistently that I ran across today on

Mind of the Innovator: Taming the Traps of Traditional Thinking
By Matthew E. May

Matthew May [...] brings our attention to the ‘Seven Sins of Solutions’, the traditional ways of thinking that prevent us from divining the most accurate—and elegant—of solutions to any problem solving situation. Using accessible examples, you’ll find yourself saying “Yes! That happens to me!” as you read. Lucky for us, May also provides methods to avoid those deadly sins and train our brains to think differently, allowing our inner innovator to flourish.

Thursday, August 2, 2007

Heading Off to DEF CON

About to head off to Las Vegas for DEF CON this morning. I'll be back on Monday. I may not be checking e-mail and certainly won't be regularly. I may simply avoid turning on my computer because there is lots going on and the network access is, well, not as safe as your mom's WiFi.