You get the idea…what was supposed to be simple (decouple something) was spinning its own Gordian knot. It seemed like a good time to see if every problem was looking like a nail (table), because all we had were hammers (MySQL).

A short search later, and we entered the world of message queueing. No, no…we know obviously what a message queue is. Heck, we do e-mail for a living. We’ve implemented all sorts of specialized, high-speed, in-memory queues for e-mail processing. What we weren’t aware of was the family of off-the-shelf, generalized, message queueing (MQ) servers…a language-agnostic, no-assembly required way to wire routing between applications over a network. A message queue we didn’t have to write ourselves? Hold your tongue.

Open up your queue…

Cutting to the chase, over the last 4 years there have been no shortage of open-source message queueing servers written. Most of them are one-offs by folks like LiveJournal to scratch a particular itch. Yeah, they don’t really care what kind of messages they carry, but their design parameters are usually creator-specific (and message persistence after a crash usually isn’t one of them). However, there are three in-particular, that are designed to be highly flexible message queues for their own sake:

• Apache ActiveMQ
• ZeroMQ
• RabbitMQ

Apache ActiveMQ gets the most press, but it appears to have some issues not losing messages. Next.

ZeroMQ and RabbitMQ both support an open messaging protocol called AMQP. The advantage to AMQP is that it’s designed to be a highly-robust and open alternative to the two commercial message queues out there (IBM and Tibco). Muy bueno. However, ZeroMQ doesn’t support message persistence across crashes reboots. No muy bueno. That leaves us with RabbitMQ. (That being said if you don’t need persistence ZeroMQ is pretty darn interesting…incredibly low latency and flexible topologies).

That leaves us with the carrot muncher…

RabbitMQ pretty much sold me the minute I read “written in Erlang”. Erlang is a highly parallel programming language developed over at Ericsson for running telco switches…yeah the kind with six bazillion 9s of uptime. In Erlang, its supposedly trivial to spin off processes and then communicate between them using message passing. Seems like the ideal underpinning for a message queue no?

Also, RabbitMQ supports persistence. Yes Virginia, if your RabbitMQ dies, your messages don’t have to die an unwitting death…they can be reborn in your queues on reboot. Oh…and as is always desired @ DigiTar, it plays nicely with python. All that being said, RabbitMQs documentation is well…horrible. Lemme rephrase, if you already understand AMQP, the docs are fine. But how many folks know AMQP? It’d be like MySQL docs assuming you knew some form of SQL…er…nevermind.

So, without further ado…here is a reduction of a weeks’ worth of reading up on AMQP and how it works in RabbitMQ…and how to play with it in Python:

Playing telephone

There are four building blocks you really care about in AMQP: virtual hosts, exchanges, queues and bindings. A virtual host holds a bundle of exchanges, queues and bindings. Why would you want multiple virtual hosts? Easy. A username in RabbitMQ grants you access to a virtual host…in its entirety. So the only way to keep group A from accessing group B’s exchanges/queues/bindings/etc. is to create a virtual host for A and one for B. Every RabbitMQ server has a default virtual host named “/”. If that’s all you need, you’re ready to roll.

Exchanges, Queues and bindings…oh my!

Here’s where my railcar went off the tracks initially. How do all the parts thread together?

Queues are where your “messages” end up. They’re message buckets…and your messages sit there until a client (a.k.a. consumer) connects to the queue and siphons it off. However, you can configure a queue so that if there isn’t a consumer ready to accept the message when it hits the queue, the message goes poof. But we digress…

The important thing to remember is that queues are created programmatically by your consumers (not via a configuration file or command line program). That’s OK, because if a consumer app tries to “create” a queue that already exists, RabbitMQ pats it on the head, smiles gently and NOOPs the request. So you can keep your MQ configuration in-line with your app code…what a concept.

OK, so you’ve created and attached to your queue, and your consumer app is drumming its fingers waiting for a message…and drumming…and drumming…but alas no message. What happened? Well you gotta pump a message in first! But to do that you’ve got to have an exchange…

Exchanges are routers with routing tables. That’s it. End stop. Every message has what’s known as a “routing key”, which is simply a string. The exchange has a list of bindings (routes) that say, for example, messages with routing key “X” go to queue “timbuktu”. But we get slightly ahead of ourselves.

Your consumer application should create your exchanges (plural). Wait? You mean you can have more than one exchange? Yes, you can, but why? Easy. Each exchange operates in its own userland process, so adding exchanges, adds processes allowing you to scale message routing capacity with the number of cores in your server. As an example, on an 8-core server you could create 5 exchanges to maximize your utilization, leaving 3 cores open for handling the queues, etc.. Similarly, in a RabbitMQ cluster, you can use the same principle to spread exchanges across the cluster members to add even more throughput.

OK, so you’ve created an exchange…but it doesn’t know what queues the messages go in. You need “routing rules” (bindings). A binding essentially says things like this: put messages that show up in exchange “desert” and have routing key “ali-baba” into the queue “hideout”. In other words, a binding is a routing rule that links an exchange to a queue based on a routing key. It is possible for two binding rules to use the same routing key. For example, maybe messages with the routing key “audit” need to go both to the “log-forever” queue and the “alert-the-big-dude” queue. To accomplish this, just create two binding rules (each one linking the exchange to one of the queues) that both trigger on routing key “audit”. In this case, the exchange duplicates the message and sends it to both queues. Exchanges are just routing tables containing bindings.

Now for the curveball: there are multiple types of exchanges. They all do routing, but they accept different styles of binding “rules”. Why not just create one type of exchange for all style of rules? Because each rule style has a different CPU cost for analyzing if a message matches the rule. For example, a “topic” exchange tries to match a message’s routing key against a pattern like “dogs.*”. Matching that wildcard on the end takes more CPU than simply seeing if the routing key is “dogs” or not (e.g. a “direct” exchange). If you don’t need the extra flexibility of a “topic” exchange, you can get more messages/sec routed if you choose the “direct” exchange type. So what are the types and how do they route?

Fanout Exchange – No routing keys involved. You simply bind a queue to the exchange. Any message that is sent to the exchange is sent to all queues bound to that exchange. Think of it like a subnet broadcast. Any host on the subnet gets a copy of the packet. Fanout exchanges route messages the fastest.

Direct Exchange – Routing keys are involved. A queue binds to the exchange to request messages that match a particular routing key exactly. This is a straight match. If a queue binds to the exchange requesting messages with routing key “dog”, only messages labelled “dog” get sent to that queue (not “dog.puppy”, not “dog.guard“…only “dog”).

Topic Exchange – Matches routing keys against a pattern. Instead of binding with a particular routing key, the queue binds with a pattern string. The symbol # matches one or more words, and the symbol * matches any single word (no more, no less). So “audit.#” would match “audit.irs.corporate”, but “audit.*” would only match “audit.irs”. Our friends at RedHat have put together a great image to express how topic exchanges work:

Source: Red Hat Messaging Tutorial: 1.3 Topic Exchange

Persistent little bugger…

You spend all that time creating your queues, exchanges and bindings, and then BANG!…the server fries faster than the griddle at McDonald’s. All your queues, exchanges and bindings are there right? Oh geez…what about the messages in the queues you hadn’t serviced yet?

Relax, providing you created everything with the default arguments, it’s all gone…poof…whoosh…nada…nil. That’s right, RabbitMQ rebooted as empty as a baby’s noggin. You gotta redo everything kemosabe. How do you keep this from happening in the future?

On your queues and your exchanges there’s a creation-time flag called “durable”. There’s only one thing durable means in AMQP-land…the queue or exchange marked durable will be re-created automatically on reboot. It does not mean the messages in the queues will survive the reboot. They won’t. So how do we make not only our config but messages persist through a reboot?

Well the first question is, do you really want your messages to persist? For a message to last through a reboot, it has to be written to disk, and even a simple checkpoint to disk takes time. If you value message routing speed more than the contents of the message, don’t make your messages persistent. That being said, for our particular needs @ DigiTar, persistence is important.

When you publish your message to an exchange, there’s a flag called “Delivery Mode”. Depending on the AMQP library you’re using there will be different ways of setting it (we’ll cover the Python library later). But the long and the short of it is you want the “Delivery Mode” set to the value 2, which means “persistent”. “Delivery Mode” usually (depending on your AMQP library) defaults to a value of 1, which means “non-persistent”. So the steps for persistent messaging are:

1. Mark the exchange “durable”.
2. Mark the queue “durable”.
3. Set the message’s “delivery mode” to a value of 2

That’s it. Not really rocket science, but enough moving parts to make a mistake and send little Sally’s dental records into cyber-Nirvana.

There may be one thing nagging you though…what about the binding? We didn’t mark the binding “durable” when we created it. It’s alright. If you bind a durable queue to a durable exchange, RabbitMQ will automatically preserve the binding. Similarly, if you delete any exchange/queue (durable or not) any bindings that depend on it get deleted automatically.

Two things to be aware of:

• RabbitMQ will not allow you to bind a non-durable exchange to a durable queue, or vice-versa. Both the exchange and the queue must be durable for the binding operation to succeed.
• You cannot change the creation flags on a queue or exchange after you’ve created it. For example, if you create a queue as “non-durable”, and want to change it to “durable”, the only way to do this is to destroy the queue and re-create it. It’s a good reason to double check your declarations.

Food for snakes

A real empty area for AMQP usage is using it in Python programs. For other languages there are plenty of references:

• Java – http://www.rabbitmq.com/java-client.html
• .NET – http://www.rabbitmq.com/releases/rabbitmq-dotnet-client/v1.5.0/rabbitmq-dotnet-client-1.5.0-user-guide.pdf
• Ruby – http://somic.org/2008/06/24/ruby-amqp-rabbitmq-example/

But for little old Python, you need to dig it out yourself. So other folks don’t have to wander in the wilderness like I did, here’s a little primer on using Python to do the AMQP-tasks we’ve talked about:

First, you’ll need a Python AMQP library…and there are two:

• py-amqplib – General AMQP library
• txAMQP – An AMQP library that uses the Twisted framework, thereby allowing asynchronous I/O.

Depending on your needs, py-amqplib or txAMQP may be more to your liking. Being Twisted-based, txAMQP holds the promise of building super performing AMQP consumers that use async I/O. But Twisted programming is a topic all its own…so we’re going to use py-amqplib for clarity’s sake. UPDATE: Please check the comments for example code showing use of txAMQP from Esteve Fernandez.

AMQP supports pipelining multiple MQ communication channels over one TCP connection, where each channel is a communication stream used by your program. Every AMQP program has at least one connection and one channel:

Code block

from amqplib import client_0_8 as amqp
conn = amqp.Connection(host="localhost:5672 ", userid="guest",
    password="guest", virtual_host="/", insist=False)
chan = conn.channel()

Each channel is assigned an integer channel number automatically by the .channel() method of the Connection() class. Alternately, you can specify the channel number yourself by calling .channel(x) , where x is the channel number you want. More often than not, its a good idea to just let the .channel() method auto-assign the channel number to avoid collisions.

Now we’ve got a connection and channel to talk over. At this point, our code is going to diverge into two applications that use that same bit we’ve created so far: a consumer and the publisher. Let’s create the consumer app by creating a queue named “po_box” and an exchange named “sorting_room”:

Code block

chan.queue_declare(queue="po_box", durable=True,
    exclusive=False, auto_delete=False)
chan.exchange_declare(exchange="sorting_room", type="direct", durable=True,
    auto_delete=False,)

What did that do? First, it created a queue called “po_box” that is durable (will be re-created on reboot) and will not be automatically deleted when the last consumer detaches from it (auto_delete=False). It’s important to set auto_delete to false when making a queue (or exchange) durable, otherwise the queue itself will disappear when the last consumer detaches (regardless of the durable flag). Setting both durable and auto_delete to true, would make a queue that would be recreated only if RabbitMQ died unexpectedly with consumers still attached.

(You may have noticed there’s another flag specified called “exclusive”. If set to true, only the consumer that creates the queue will be allowed to attach to it. It’s a queue that is private to the creating consumer.)

There’s also the exchange declaration for the “sorting_room” exchange. auto_delete and durable mean the same things as they do in a queue declaration. However, .exchange_declare() introduces an argument called type that defines what type of exchange you’re making (as described earlier): fanout, direct or topic.

At this point, you’ve got a queue to receive messages and an exchange to publish them to initially…but we need a binding to link the two together:

Code block

chan.queue_bind(queue="po_box", exchange="sorting_room",
      routing_key="jason")

The binding is pretty straight forward. Any messages arriving at the “sorting_room” exchange with the routing key “jason” gets routed to the “po_box” queue.

Now, there’s two methods of getting messages out of the queue. The first is to call chan.basic_get() to pull the next message off the queue (if there are no messages waiting on the queue, chan.basic_get() will return a None object…thereby blowing up the print msg.body code below if not trapped) :

Code block

msg = chan.basic_get("po_box")
print msg.body
chan.basic_ack(msg.delivery_tag)

But what if you want your application to be notified as soon as a message is available for it? To do that, instead of chan.basic_get(), you need to register a callback for new messages using chan.basic_consume():

Code block

def recv_callback(msg):
     print 'Received: ' + msg.body
chan.basic_consume(queue='po_box', no_ack=True,
                callback=recv_callback, consumer_tag="testtag")
while True:
     chan.wait()
chan.basic_cancel("testtag")

chan.wait() is looped infinitely, which is what causes the channel to wait for the next message notification from the queue. chan.basic_cancel() is how you unregister your message notification callback. The argument specifies the consumer_tag you specified in the original chan.basic_consume() registration (that’s how it figures out which callback to unregister). In this case chan.basic_cancel() never gets called due to the infinite loop that precedes it…but you need to know about it, so it’s in the snippet.

The one additional thing you should pay attention to in the consumer is the no_ack argument. It’s accepted on both chan.basic_get() and chan.basic_consume() and defaults to false. When you grab a message off a queue, RabbitMQ needs you to explicitly acknowledge that you have it. If you don’t, RabbitMQ will re-assign the message to another consumer on the queue after a timeout interval (or on disconnect by the consumer that initially received it without ack’ing it). If you set the no_ack argument to true, then py-amqplib will add a “no_ack” property to your AMQP request for the next message. That will instruct the AMQP server to not expect an acknowledgement for that get/consume. However, in most cases, you probably want to send the acknowledgement yourself (e.g. you need to put the message contents in a database before you acknowledge). Acknowledgements are done by caling the chan.basic_ack() method, using the delivery_tag property of the message you’re acknowledging as the argument (see the chan.basic_get() code snippet above for an example).

That’s all she wrote for the consumer. (Download: amqp_consumer.py)

But what good is a consumer, if nobody is sending it messages? So you need a publisher. The code below will publish a simple message to the “sorting_room” exchange and mark it with the routing key “jason”:

Code block

msg = amqp.Message("Test message!")
msg.properties["delivery_mode"] = 2
chan.basic_publish(msg,exchange="sorting_room",routing_key="jason")

You may notice that we set the delivery_mode element of the message’s properties to “2”. Since the queue and exchange were marked durable, this will ensure the message is sent as persistent (i.e. will survive a reboot of RabbitMQ while it is in transit to the consumer).

The only other thing we need to do (and this needs to be done on both consumer and publisher apps), is close the channel and connection:

Code block

chan.close()
conn.close()

Pretty simple, no? (Download: amqp_publisher.py)

Giving it a shot…

Now we’ve written our consumer and publisher, so let’s give it a go. (This assumes you have RabbitMQ installed and running on localhost.)

Open up the first terminal, and run python ./amqp_consumer.py to get the consumer running and to create your queues, exchanges and bindings.

Then run python ./amqp_publisher.py “AMQP rocks.” in a second terminal. If everything went well, you should see your message printed by the consumer on the first terminal.

Taking it all in

I realize this has been a really fast run through AMQP/RabbitMQ and using it from Python. Hopefully, it will fill in some of the holes of how all the concepts fit together and how they get used in a real Python program. If you find any errors in my write-up, I’d very much appreciate it if you’d please let me know (williamsjj@digitar.com). Similarly, I’d be happy to answer any questions that I can. Next up….clustering! But I’ve got to figure it out first.

NB: Special thanks to Barry Pederson and Gordon Sims for correcting my understanding of no_ack’s operation and for catching syntactically incorrect Python code I missed.

NB: My knowledge on the subject was distilled from these sources, which are excellent further reading:

• zeromq: Message-oriented Middleware Analysis
• RabbitMQ .NET Client Library User Guide 
• Advanced Message Queuing Protocol: Protocol Specification Version 0-8

• (Good) National 3G (U.S.A.) coverage
• Minimum top end throughput around 1Mb/s
• ExpressCard form factor (nothing sexier than a wrist-sized dongle cantilevered off your USB port)
• Support for Mac OS X

Folks that know me probably are stunned at the last one. As of April 29th I kicked the Dell habit. My regular target of abuse is now a MacBook Pro. But that's a whole other story…

Anywho, those req's really narrowed it down to two players: AT&T and Verizon. Both offer national 3G access at speeds of 1Mb/s or greater. But they take two different approaches to it…

HSPA (High Speed Packet Access)

High Speed Packet Access is really the joining of two different 3G GSM protocols: HSDPA and HSUPA (the D and the U are “downlink” and “uplink” respectively). On AT&T's network, HSPA should give you average speeds around 1.8 Mb/s down and 800 Kb/s up. My experience has been that this is true across their network…as long as you can get a 3G signal. In fact, in some areas (LA and San Antonio) it wasn't uncommon for me to get around 2.2-2.5 Mb/s down. With tower upgrades coming early next year, the downlink speed should boost further to about 7.2 Mb/s. Overall, pretty darn good for no leash. Factor in the fact that HSPA is a 3G GSM standard widely deployed across Europe/Japan and suddenly you've got a great data solution worldwide (an issue given some upcoming trips). Oh, I forgot to mention…some places in Europe have already deployed 14.4Mb/s HSDPA (HSUPA deployment is somewhat spottier).

Compared to EV-DO, HSPA also has some design advantages. For example, both EV-DO and HSPA time slice transmission to connected clients, but HSPA can transmit to 10 clients in single time slice, whereas EV-DO can only transmit to one client per time slice. Also, HSPA towers possess the capability to figure out which clients have the best signal quality and will transfer bandwidth capacity from clients who can't use it (bad signal) to clients that can (excellent signal). Of course, even with all of its advantages, the HSPA network is being run by AT&T…and they could screw up implementation of a PB&J sandwich…

EV-DO (EVolution – Data Optimized)

Like HSPA, EV-DO is a CDMA-based 3G protocol. Unlike HSPA however, it is not a GSM body standard and is instead the successor to CDMA2000. So, outside of the U.S.A, Korea, areas of Japan and piroshki stands in the former Soviet-bloc you're pretty much out of luck for access. However, it does provide 1Mb/s speeds regularly. Upload speeds are in the 200-500 Kb/s range.

With that brief understanding, I motored down to the Verizon and AT&T stores and picked up service with both companies (AT&T and Verizon have 30-day refund and new service cancellation policies).

Behind door number 1…

For a couple of years now, I've heard phenomenal things about Verizon's BroadbandAccess (EV-DO) service. People seemed to rave about it's coverage and reliability…and they're right. Verizon's biggest plus is it's consistency. It may not be as fast or have as low latency when AT&T is on the ball, but they'll deliver the same service levels every time you power up. I don't care if I was in Boise, LA, or San Antonio, Verizon delivered 800-1000 Kb/s throughput and 230ms latency like clockwork. Sometimes it was a bit better or a bit worse, but only by about 10% (exception was the trip up the coast to Malibu where Verizon dropped down to 2.5G service and AT&T was nowhere to be seen).

The other nice thing about Verizon is the Novatel V740 ExpressCard. It has excellent support in OS X. Pop it in and OS X's built-in WWAN manager configures the card, activates it with Verizon and away you go. No special software to install. You even get a nice little signal strength meter on the task bar (yeah…yeah…taskbar…Windows habits die hard ).

The gal you really wanna take home to Mom…

I wanted AT&T to be the best…honest. I'm a current AT&T Wireless voice customer and love the phones, the stores and the service. However, 3G service with AT&T lacks Verizon's consistency. Initially, 3G coverage could be hit and miss when I started 4 weeks ago. However, their big push for blanket 3G coverage in advance of the 3G iPhone launch has improved the 3G network dramatically in the last week. Although the coverage is spot on now, the service level is not in terms of latency. Throughput however is phenomenal. 1500-1800 Kb/s downlink speeds over 90% of the time, with solid 2200-2500 Kb/s in areas with the latest tower gear. So for the majority of applications, AT&T LaptopConnect is a superior solution to Verizon. But…not for me.

Quite a bit of the remote work I do involves either SSH or Windows Remote Desktop over VPN. There's few things more annoying than mistyping a command and waiting for the refresh to catch up so you can go correct it. As a result, better latency means a happier camper 'round these parts. That's not to say that AT&T's latency is awful. In fact, it's better than Verizon about 80% of the time when you measure it. So why am I complaining about it? Well, 150ms latency is only good if it stays at 150ms. AT&T's deployment of HSPA causes latency spikes regularly, particularly under load. As a result, I started doing an combo test on both services…load a YouTube video and concurrently check the ping over a VPN tunnel. If you try it, you'll see both Verizon and AT&T's latency spike dramatically. Hmm..you're probably thinking, “so AT&T is better than Verizon both with and without heavy load…why won't you say its better?”. Because, it doesn't feel faster. It was really hard to put a metric on this, because while the measurements were better on AT&T, the lag while typing on an SSH connection always felt a little (to a lot) bit slower. In fact, I kept reminding myself that this had to be in my head, because the ping measurements were better than Verizon. Then while in San Antonio I tried using Skype.

San Antonio expectedly has the best coverage of any AT&T area I've been in. Consistent throughput above 2200 Kb/s and latency below 150ms. So imagine my surprise when my SSH sessions seemed laggy, and the Skype calls would start great and then break down within about 3-4 minutes. You could hear the person on the other end of the call fine, but they started having issues hearing me and my video would lock up for them. If you turn the video off it'd buy you another 4-5 minutes before the call went haywire. So back in went the Verizon card. Bang. Perfect SSH sessions. Crystal clear call quality on Skype…and the folks on the other end said not only was the video smooth but the quality of the picture was better (Skype must adjust video quality based on connection quality). A 45-minute Skype call completed with no audio or video issues on Verizon.

I tried the Skype exercise about 3-4 times over a 48-hour period with the same results. Every time I'd give AT&T a shot, and every time I'd have to drop in the Verizon card to complete a decent conversation. This bodes not well for the rumor that the 3G iPhone will take advantage of HSUPA for video conferencing. On the positive side, those consistent 2200 Kb/s AT&T downlink speeds meant I was able to suck down the OS X 10.5.3 system update (420MB) in about 30 minutes (~1500 Kb/s sustained average).

The other major issue with AT&T is the Option GT Ultra Express card. On the positive side, it supports HSUPA so you can take advantage of fast uplink speeds. Unfortunately, it isn't supported natively by the OS X WWAN subsystem (unlike it's unavailable predecessor, the Option GT 3.6 Express which is natively supported). So you have to install Option's GlobeTrotter software, which isn't a slick as the native support and frankly feels poorly built. A lot of folks on the Apple and AT&T forums have also complained about GlobeTrotter frequently crashing for them. To some degree I suspect the inconsistent performance I get from AT&T (despite the metrics) might be due to GlobeTrotter. There's also Launch2Net by NovaMedia, which provides 3rd party drivers for the GT Ultra Express. Still not native, and amazingly Launch2Net axes the native WWAN utilities that the Verizon card leverages (Launch2Net got uninstalled faster than Vista on a 286). Supposedly, OS X 10.5.3 was going to include native support for the GT Ultra Express, but as of 10.5.3's release yesterday…no dice.

Lastly, there's price. Both AT&T and Verizon charge $60/month. However, AT&T's service is unlimited where Verizon's service is 5GB/month (and $0.49/MB over that).

End of the road…

So where does that leave us? If you need reliable latency and pretty darn good speed, Verizon is your best bet in my opinion. On the other hand, if the majority of your remote work involves the web, e-mail or anything else that's not latency sensitive, AT&T is far superior and will allow global roaming. Frankly, I'm kind of anxious to hear from someone who has the GT Ultra Express on a Windows machine to find out if the inconsistent performance I experienced was specific to GlobeTrotter for Mac. Personally, I'm going to keep both services. There were a handful of times that Verizon's latency was abysmal, but AT&T's was great. Enough that I realized in an emergency I'd really need to have the option of either service.

Here's hoping AT&T's 3G latency improves…and that Apple gets with the program and includes native support for the Option GT Ultra Express…the 3G ExpressCard of choice for Apple's carrier of choice. Sorry that this post blathered on a bit long. I hope this saves other folks from having to do this much evaluation legwork.

(Here is the XLS sheet with observed metrics for both services: AT&T vs. Verizon Benchmarks )

PDF
Slideshare

Overall, the Web UI is very easy to navigate and options are not buried more than 2 clicks deep. However, there are two areas where the ACOS Web UI is absolutely a pain in the rear:

• Grid-metaphor editing.
• Heinous layout for the relationship between physical interfaces, VLANs and virtual interfaces.

Grid Editing 

One of the most common day-to-day tasks we end up doing with a load balancer is enabling/disabling a batch of real servers for upgrade. Generally, we want to:

1. Disable real servers A, C, and E. (Leaving B & D enabled).
2. Upgrade A, C, and E.
3. Swap A, C, and E back into battery and take B & D out.
4. Upgrade B & D.
5. Put B & D back in.

This is a perfect application where you want to be able to pull up the settings for multiple entries in one edit table. With the settings for real servers A,B,C,D, and E up on the same page, you can change all of the applicable settings all at the same time, verify each server is correct, then bam!…slam the new settings into place all at once. Unfortunately, this is not possible with the ACOS Web UI. The only thing you can do to multiple entries at once is delete them:

But simple maintenance of real server status is not the only place with the table editing metaphor is helpful. It is indispensable when trying to balance which VLANs are on which physical ports. Having to drill into an entry, make the change, and then re-examine the grid view to see how it looks is very tedious. It's much easier to pull up all the necessary interface/VLAN assignments on one view, edit them in-place and then apply them with a single-click once they look right. It seems that the goal of any good Web UI should be  to minimize round trips and enable batch application as much as possible. This was an area where the Nauticus/Sun Web UI was phenomenal. Any grid view could be turned into an edit table. On the other hand, if you only selected one entry to edit, the Nauticus Web UI was smart enough to reformat the one entry into a single column of editable values (so it fit horizontally without scrolling). Quickly swapping batches of real servers in and out of service is not a task we're looking forward to with the AX2200.

Network Relationships & Just Being Friendly 

This is not an uncommon metaphor for dealing with VLANs and the IP interfaces that sit on them:

• VLANs entries belong to physical interfaces.
• Virtual IP interfaces are created and belong to specific VLANs entries.

To A10s credit, it's a familiar metaphor that is instantly accessible, and they even kept the ve0, ve1 virtual interface naming convention that's common to Cisco and Foundry equipment. Where they went wrong is not making it easy to tag a friendly name onto the VLAN and virtual interface entries.

What's the purpose of VLAN 1234? Well it's attached virtual interface ve0…that's helpful. What on God's Green Earth does ve0 serve? You can't tell easily from the VLAN page. You either have to dig out your documentation, or open the virtual interface list in a separate window:

The simple solution on Foundry and Nauticus/Sun gear was what you could call “friendly names”: A simple user description for each VLAN, interface and virtual interface. Can't remember what VLAN 1234 does…no problem…it's friendly name says “tier1_realservers”. Oh! That's right, VLAN 1234 contains the application servers for tier 1 of our application and ve0 is the virtual interface that serves that subnet. Toggling back and forth between tabs in Firefox for VLANs and virtual interfaces while setting up the test AX2200 has been a barrel of monkeys. Frankly, “friendly” or “vanity” names should be able to be attached to any type of entry whether it's a real server, a physical interface, or an SSL certificate.

Other nits so far: 

• Appliance will not boot if hard drives are not in exactly the same slots as shipped (not expected for a RAID-1 setup).
• Can't find a mechanism in the Web UI to generate a CSR.
• Can't find a way to import a PEM file (Must import certificate and key file separately.)
• There doesn't appear to be a way to load certificates and keys by pasting them into a text box.
• Host name is not notated at the top of the GUI and in the page title at all times to help identify which box you're in.
• Virtual interfaces that are already in use still show up in the VLAN creation screen as assignable. Only when “Apply” is clicked does a JavaScript alert box tell you there's an issue.
• Physical front panel status light only blinks when there's a problem. Does not turn amber or red. Very unnoticeable if you don't already know there's an issue.
• Showing system interfaces via the CLI is “sh int” instead of “sh sys int” on Foundry gear.

The last one is not something normally you'd complain about. All networking vendors seem to do it differently. However, given the fact that A10 is staffed with so many ex-Foundry Networks folks, and the fact that the ACOS CLI is identical to Ironware in so many areas, it's an unwelcome surprise when “sh sys int” errors out while you're in the CLI.

Needless to say, we're still talking about the AX2200, so we're fairly happy with what we've seen so far. However, “friendly naming” and table editing  really need to be fixed in an upcoming version of ACOS. The current way of doing things is probably only acceptable in very small environments where the boxes don't get touched very much. This weekend is dedicated to SLB testing…so hopefully more advanced configuration is where the Web UI really comes together.

That's all that's fit to print as they say.

Technorati Tags: A10 Networks, ACOS, AX2200, DigiTar, Sun

But what blew us away was a very simple feature we'll call “assignable virtual IP address (VIP)”. Assignable VIP functionality allows you to create two virtual load balancers (internal and external) with no routing in common, and attach your real servers to one (internal), while advertising the VIP on the other (external). Because there is no routing path between them (all traffic hitting the VIP is essentially memory copied to the internal load balancer for SLB processing), no servers sitting in your DMZ can compromise or talk directly to your real servers. They simply can't talk to something that there's no routing path to. As a result, you have a separate clean management path to your real servers that is entirely inside your trusted network, and incredibly simplifies your topology (no ACLs!). It is by far the best application of virtualization in a network device we've ever seen. However, the halcyon days came to an end in April of 2007 when we were informed that Sun intended to EOL the entire N series and shutdown the load balancing group they had acquired with Nauticus. Given that there were no other products on the market in April of 2007 that could even remotely drop seamlessly into our new topology, we decided  to wait and see what Sun might do next.

A year later not much has changed, and Sun still doesn't have a coherent strategy on load balancing to replace the N series. While our units would continue to be supported for the next 5 years, there won't be software updates, and definitely no updates to the phenomenal FPGAs that make the box scream. There are flaws in the N series that need bug updates…things that would be livable if they were going to be fixed. But in a production environment no bug fixes is simply not an acceptable strategy. So we're back in wonderland…

To cut to the chase, we talked with all the major vendors and settled down to F5, Citrix/NetScaler, and Cisco. Only Cisco, with their ACE platform, has any virtualization story whatsoever. Everyone else has no virtualization plans that they're telling their sales dudes about. All 3 can cobble together an inelegant and obfuscated configuration to allow us to maintain our topology and security stance, but none can do the “assignable VIP” magic that made Sun/Nauticus such an amazing application of virtualization and so clean to administer.

In the middle of all this, a trusted friend at Sun recommended we take a look at a new load balancing company, A10 Networks. Now A10 doesn't have virtualization in their platform today, and they definitely don't have “assignable VIP”. But they have a story and roadmap that will make any Sun/Nauticus customer get a big silly grin on their face. You'll have to talk to A10 to find out the particulars.  

What does A10 have? A phenomenal architecture on paper, and sane licensing. While FPGAs are what made the Nauticus design scream, being entirely FPGA and ASIC driven was also what drove the cost of bug fixes up. It was difficult for them to add L4/L7 features at the same rate that F5 and others were, because it usually required a modification of the FPGA layout. Enter what appears to be a brilliant design compromise and excellent capitalization on the Intel/AMD race for core count. The A10 AX2200 and above have L2/L3 ASICs, SSL ASICs,  and a L4/L7 traffic director FPGA. The FPGA dynamically assigns new connections to each of the box's 4-8 Xeon cores for full L4/L7 processing. Also, each core operates independently from the others. That is to say, there is no contention or synchronization penalties for using more cores. Add more connections and the traffic FPGA evenly distributes them among the cores, and stitches the results back together for the client. Near perfect parallelization. All of the heavy L4/L7 lifting is done entirely in software on generic Xeon cores. This allows A10 to quickly add the complex features (like F5) that would have required an FPGA modification on the Nauticus gear. The excellent parallelization model ensures the performance hit encountered by using generic CPUs instead of FPGAs can be made up for linearly by adding buckets of cores. The FPGA is therefore much simpler in design than what Nauticus required. But as I said, this is all on paper.

However, it is an equally seductive design to what Nauticus created. F5, NetScaler and Cisco all have L2/L3 ASICs in their boxes but nothing really significant in terms of hardware acceleration in the L4/L7 areas (F5 does have their L4 ASIC that does provide good acceleration of basic L4 TCP termination load balancing). So we've decided to leap again and take a chance with A10. Also, A10 includes Global Server Load Balancing for free and does not engage in F5's hideous practice of licensing HTTP compression and SSL offload capacity by the MB/s…oh and A10 has TCL-based aRules.

So we eagerly awaited the FedEx guy on Thursday to deliver our new pair of AX2200s for validation testing. With a 100lb thump they landed solidly on our testing table, and a couple flicks of a box cutter later…

A10 Networks AX2200 – Front Panel

What came out of the box looked like the unholy progeny of a Sega Master System and the portholes from a Buick Roadmaster. Needless to say she ain't a looker. Frankly, at this price level the gear should be drop-dead sexy. Yes, it may be shallow, but its a requirement when you're trying to justify an $80 grand list price. To add insult to injury, the portholes don't serve any utilitarian purpose like cooling…they're actually a solid piece of plastic. As a counter example, the N2120 and Sun's standard 2U server design are phenomenal:

Sun N2120 & Sun 2U Server – Front Panel

They exude the simplicity and power that's concealed inside…a little glimpse to the upper echelons of what you're spending the company's hard earned bananas on. But what the AX2200 gets right is spot on build quality. It's solid with no rattles. The power supplies slide smoothly and easily. Re-seating a supply gives a firm click and solidly locks them from removal. Overall, it's downright Teutonic in construction. Sort of like an older Audi S8, built to run forever like greased lightning, but not much to look at. A10 could take Audi's cue and start paying attention to creating looks that match the engineering.

A10 Networks AX2200 – Back Panel

One very nice feature of the AX2200 for a load balancer is the hot swap fan tray. Not having to spirit the whole unit back to Boston because a fan went South is a nice change from the N1216. Also, the interior build quality is just as clean and professional as the exterior components. Hard edge connectors and system board tracings are used almost entirely, with nearly no ribbon cables cluttering up the interior. Only nit is the front management NIC is run to the motherboard via an RJ-45 cable routed to the back. Don't let the server exterior of this box fool you, this is a purpose built system with specialized ASICs and FPGAs inside.

A10 Networks AX2200 & Sun N2120 – Side View

As with any new appliance, this one has a couple of strange design foibles that go deeper than its looks. First, the box vents in from the sides and exhausts out the back. In that regard, its neither quite at home in a rack with your servers or with your switching and routing gear. The strange intake flow means if you rack the AX2200 above your side-to-side vented switching gear, you'll likely overheat the AX2200 as it sucks in the switches' side exhaust air. Luckily, we have some Juniper kit that vents front to back, so we will likely rack the AX2200s with them. Also, the locking drive carriers are a bit frustrating. It's a nice feature that they can be locked, but inserting the key with any more force than a gnat breaking wind pops out the removal handle. It's obviously an off-the-shelf carrier that no designer actually tried before spec'ing it out of the part book.

A10 Networks AX2200 – Front Panel No Bezel

On the positive side, the serial connection is on the front and is a Cisco-style RJ-45. Yippeee! No RS-232-to-rollover adapters to hook it into our Dominion SX! It may seem like a small thing, but it really means fewer parts to lose, break and stock at the data center. I wish I could say they had the foresight to also put a sticker on the front with the box's serial number…but unfortunately not so much. You'd better note the serial number before you rack the AX2200, otherwise its going to be crane and strain time to see the sticker on the bottom of the unit. Scratch that…the serial number is conveniently placed on the rear left of the unit as well. It's not as easy to see as the front given the Also, they did show the company's Foundry Networks pedigree by shipping a very Foundry Networks-esque self-test sheet with the unit:

AX2200 Self-Test Slip  

Kudos for the self-test paperwork. If you keep that on file, you can probably forgive the serial number sticker's ill fated position on the unit's underside.

Overall, first impressions…the construction and major design decisions are terrific. This box looks the part internally, and feels the part externally as a major piece of core infrastructure. Next step is to rack her and beat the heck out of her with our test rig: a screaming UltraSPARC T2. WIll post more soon on how the AX2200 stands the scorching…

Technorati Tags: A10 Networks, ACOS, AX2200, Sun

1. No client/server settings per port! Hooray! The Alteons (even the 2424s) inherited from the Alteon AD4s and 184s the need to enable client and/or server processing per port. For those who are not familiar, server load balancing basically can be reduced into two operations:
• Client processing: When a packet comes in from a web browser to the web switch, its header has a TO fieldthat's the IP address of the web switch, and a FROM fieldthat's the IP address of the web browser. Once the web switch gets the packet and decides which back-end server to send it to, it has to replace the packet's TOwith the IP address of the back-end server. If the web switch didn't change the TOand simply sent the packet on, the server would ignore the packet. Sort
  of like receiving a letter addressed to somebody you don't know. So in a nutshell, server processing is simply replacing the web switch's IP address with the selected back-endserver's IP address in packets from the client.
• Server processing: When the back-end server decides to send a response packet back to the client, the reverse of server processing has to occur. If the web switch were to simply send the packet from the server back to the client without client processing, the client would ignore the packet. Why? Well, the client sent the packet to the IP address of the web switch and expects a reply from that IP address, not the server's IP. It sort of like sending a letter to Aunt Gertie, but getting the reply
  from Aunt Gertie's nurse Josie. You don't know who Josie is, so you toss the reply thinking its junk mail. Client processing fixes this by rewriting the FROM in the server's reply to the IP address of the web switch.
• An Alteon is a bit unusual in that instead of one massive SLB processor it has 8…one per port (this is fixed in the 2424s, but they imitate the older behavior for backward compatibility). So if you have one port connected to your servers and a second port connected to the Internet, you have to enable Client processing on the Internet-facing port and Server processing on the server-facing port. The reason is that the 8 individual processors aren't bulky enough to do BOTH the client and server processing. As
  a result, the operation gets split between ports in a way you specify. So you have to remember which kind of processingis which, and set it appropriately on the right ports. This is a MAJOR pain in the butt. If you get client and server processing confused and set a port to the wrong one, load balancing just isn't gonna work for you today.
• The SunFish don't have this limitation. They just make it work. Concentrate on creating your VIPs and RIPs and the rest is taken care of for you. Its really a spectacular change for us! It was so easy, that it wasn't until I was driving home that it struck me I hadn't had to fool with client or server processing at all.
2. XML-over-HTTP! As I was complaining about the lack of a heads-up-display on the SunFish, I ran into a very cool feature!On most of the pages that list settings or statistics in the SunFish WebUI, there's a little button labeled “XML”. If you click on it, you get the settings or stats you were looking at…but XML encoded! This means you can write your own scripts to consume the status of the SunFish! All your program needs to be capable of is downloading pages via HTTP, and consuming XML.
   The upshot is that this feature enables us to write our own stop gap heads-up-display. Its much simpler than messing around with SNMP calls and the like. Particularly, given our familiarity with consuming web services. This is a terrific feature! Props to the SunFish team for providing an XML interface to the unit. Simply amazing.

Technorati Tags: N1400V, Sun

Might as well call it CryptoFish…

Up until this point, we've never had the pleasure of using SLB-based SSL-offload. We've just trudged along…scaling the ol' SSL proxies along with the app servers. While cost-effective, the customer doesn't get Speedy Gonzales latencies, and the number of moving parts goes up. As you might guess from yesterday's entry, a main design goal for us is reducing moving parts. DigiTar believes y'all can't break stuff that doesn't exist. Needless to say, moving to SSL-offload in our load-balancers is something we're pretty keen on. Among other things, we'll be able re-purpose all those beefy SSL proxy servers for other things…like space heaters.

Enter today's first chore…migrate the HTTP SLB VIP to an HTTPS config leveraging the SSL-offload capabilities of the SunFish. While we've never configured SSL-offload before, given this review on the Alteon 2424-SSL, I was prepped for beastly day. In case you don't want to read the link, on a 2424-SSL, you're looking at a configuration task that involves redirection and 3 sets of filters on 3 different VLANs. This is all to get the traffic to and from the SSL card…which is more of a suckerfish (yeah the ones that feed on sharks) than a real part of the box.

An Alteon 2424 with its SSL card in tow…

So what's all this leading up to? We had SSL up on the SunFish in about 5 minutes! That's right, four clicks of the mouse and bang I was done. What's more…it worked! The hardest part was trying to pull the proxies' key and cert out of the Subversion repository. Here's the 4 steps to converting an existing SunFish HTTP SLB group to utilize SSL-offload goodness:

1. Copy your existing key from its Apache PEM file, and paste it into the import page on the SunFish WebUI. Oh…you've also got to name it something clever. Alright, maybe not clever, but its gotta have a name.
2. Grab the certificate portion from the aforementioned PEM, and paste into the import page on the SunFish WebUI…and select the name you assigned the key in the last step using the drop-down.
3. Delete the existing HTTP virtual service.
4. Re-create the HTTP virtual service as an HTTPS virtual service with all the same parameters as before, only this time select your cert and key from the “Certificate And Key Name” drop-down.

Its that easy. No redirection filters. No funky hidden VLANs. No nothin'…no kidding. There was so much time left-over, I spent 20 minutes figuring out how to do on-the-fly HTTP header modification…which also worked perfectly. The SunFish is so easy to use for crypto, its purchase price is probably justified by what we'll save doing things other than configuring Apache SSL proxies. If the 2424 wasn't dead on our purchase list by now, this was certainly a double-tap to its mainboard.

You need GigE?

Well we tried to setup redundancy…but the effort was a bit stillborn. One of the only complaints I've got about the physical aspects of the SunFish is that all of its ports are SFPs. As a result, to use RJ-45 cables you have to use GigE copper SFPs (which David C was very generous to provide). The problem is that an SFP has no concept of 10/100/1000. Its designed to be a gig optical port. As a result, we're pretty limited in our dev server room as to what we can hook it up to. In terms of the test, the X4100s are directly connected (they've got GigE ports), and the “Internet” port goes to a GigE-capable firewall we use. Yeah…I know. Its weird. We have a GigE firewall but no GigE switches in our dev lab. Alas, that means we can't test the redundancy…but that doesn't mean I can't pontificate about it.

The SunFish definitely has Alteon-rising when it comes to redundancy. No elegant custom protocol that binds two units as one. Rather, they use VRRP to tie the interfaces of redundant SunFish together, and an unnatural progeny of VRRP they call VSRP to handle failover of SLB virtual services between units. All-in-all darn identical to the Alteon…almost.

Like the Alteon, VRRP has to be set up separately on each interface you want failover for. That's a royal pain in the keester. However, VSRP is a little different, and lot better than the Alteon's way of hacking VRRP to handle virtual service failover. With the Alteon you have to configure a VRRP instance for each SLB virtual service…just like the interfaces. When you're running close to 15 virtual services per web switch, this is beyond a royal pain. Its excruciating. Rube Goldberg could design a better way.

The SunFish does it much better. You just turn VSRP on. Yes…that's just about it. Of course, you have to set which unit in a pair is the master and which is the backup…but that's all you're required to do. No matter how many virtual services across a disparate number of vSwitches might catch your fancy, you only have one VSRP instance to enable per web switch. For us that's incredible! As much as I would prefer a custom protocol that makes the entire process transparent, I'll take a SunFish thank you very much.

Feelings thus far…

So far I've been incredibly impressed by the SunFish. It is by far the most robust and powerful web switch a kid with a shiny nickel can buy (OK…500,000 shiny nickels)…without buying its big brother. There isn't one thing so far that wouldn't allow the SunFish to surpass our needs in the following areas:

• Consolidating a gaggle of web switches down to 2. Securely.
• Putting a brigade of SSL proxy servers out of commission.
• Giving us more power than we'll ever need in a single rack. Heck, this baby will run multiple processing silos for us without breaking a sweat.

The one area I can't really stress enough is the amount of flexibility the SunFish will give us. The raw power combined with the virtualization capabilities will only increase our ability to deliver mind-blowing solutions to our customers (and keep the price lower than our competitors). Now the task is to put these boxes into a more production environment. More on this soon.

P.S.
We still want our AlteonEMS for SunFish! This would make the SunFish far and above the best web switch period. Yeah F5 may have its iRules…but you put a few of those on one of their boxes and the gear keels over. To meet our needs with F5 gear we'd need to buy enough to finance a fleet of Ferraris. SunFish or Alteons baby. Power over looks.

Technorati Tags: N1400V, Sun

(the N1400V was codenamed SunFish, which is infinitely cooler ).

A little background:

DigiTar uses load balancers like a Frenchman uses perfume…everywhere and often. Given our predilection for them, we need the most powerful web switches we can get our hands on. They're the sinew that holds us together. Up until now we've been deploying pairs of Nortel Alteon 184s everywhere we needed load balancing, which has included our MySQL fail-over strategy (which the SC project is supposed to replace…ugh!). Our Alteons have been rock solid, and frankly, are my favorite pieces of gear. They work…all the time…every time. We've truly abused them, and they keep humpin' it up the mountain… If you need a load balancer, you can't go wrong with an Alteon and I can't say enough nice things about these bad boys.

Unfortunately, going back to “needing the most powerful ones”, the 184s are starting to get tapped out. Also, we'd like to consolidate down into fewer of them. So just about the time the SunFish arrived, we were getting ready to replace our 184s with a fewer number of Alteon 2424-SSLs. (If you've never seen an Alteon 2424 do its magic…you're missing out. Its a beast! Tapping it out is a challenge.) Alas, our local Sun evangelists asked if we'd looked at the Sun load-balancers…and thus like so many of our odysseys of late, began our ascent into another Sun journey of discovery…(thanks Jamison & Elizabeth!)

Its only a phone call…

Like every one of our technology disruptions, this one began with a tiny little con call…what could it hurt? Right? On the other end of that call was David C. David put up with every single pushy (and somewhat Alteon-bigoted) question we had. At the end of the phone call we were a bit intrigued, but it was the presentation numbers after the call that sold us on ripping out our beloved Alteons.The fact that we've been told by our “sources” that a lot of the original Alteon engineers went to Nauticus (pre-Sun) didn't hurt.

So what sold us? Well, the L4-L7 load balancing throughput was more than 3x what we were expecting from the 2424-SSLs, and the SSL acceleration throughput was so much higher I can't even mention it without embarrassing Nortel. If the 2424-SSLs are beasts, then the SunFish are 8000lb silverbacks on a steroid-regimen that would make Barry Bonds permanently sterile. And the REALLY ridiculous part… the SunFish (N1400) are the babies of the line. There is an N2120 with twice the performance.

Outside of the performance, what really convinced us to attempt a heart transplantwas the ability of a SunFish to slice itself into 10 virtualSLB switches. One thing we had tried to do early on was consolidate multiple SLB groups into a single Alteon switch. The problem was security. Because the Alteon (like the SunFish) is first and foremost a switch (the source of its power), it is almost impossible to segment SLB groups on secure subnets from SLB groups on insecure subnets. Even using VLANs we've occasionally seen packet leakage in testing. So here was what was buzzing around in our brains:

1.) We want to consolidate down to fewer web switches (load balancers).
2.) Secure subnets have to be absolutely segregated from insecure subnets.
3.) One SunFish pair could easily replace 5 of our Alteon pairs.
4.) SunFish can slice themselves into completely separated vSwitches.

hmmm….I wonder….could we collapse down to a single pair of SunFish per facility?

As with many things in DigiTar's history, Providence has introduced what we didn't know we needed at the precisely right time…

Enough already…where are those first impressions?

I'm running out of time tonight…so here's a quick run-down (there'll be more, I promise):

• SunFish are truly…completely…utterly different animals from any other web switch. The concepts of vSwitches make it necessary, but boy is it worth it.
• Any schlocker can configure the whole thing from an incredibly slick Flash WebUI…and the CLI ain't half bad. It isn't the Alteon CLI but, hey, nothing 'cept JUNOS has a better CLI than Alteon's WebOS.
• Its a little harder than it should be to find the documentation. The SunFish OS 3.0 documentation is spotty as all get out. If you can configure this puppy from 'em, there's $20 waitin' for ya. The trick is to grab the System Configuration Guide for the N2000 OS 2.0. The N1000 and N2000 series run the same OS, and thankfully the syntax has stayed the same from 2.0 to 3.0.
• The System Configuration Guide for the 2.0 OS is a gem. Its crystal clear and logical. If you have a firm grasp ofSLB fundamentals, this is all you need. To be honest I can't say enough about this guide. Compared to the steaming piles of cow-manure that are the WebOS guides, the N2000 guide really blows ya out of the doors. Nothing stinky about it…even the English is crisp…smells of starch.
• Alright…I still can't shut up about the guides. Thanks Sun for not making us go to gold-plated classes to configure this thing. Nortel…well you kind of suck about that…but that was your intention no?
• Inside of 12 hours I had the guide read, my development servers configured for the test, and the web switch configured and running properly. That's pretty incredible to me. It took weeks of pain and suffering to get the first Alteons up over 2 years ago. Considering that this was a much more complicated setup than that one two years ago, its pretty amazing. Its a testament to the guides. Keep in mind that in both cases (the Alteons 2 years ago, and the SunFish today), there was zero help or training involved outside of the printed word.

• Test Config:

• 1 SunFish load balancer.
• 1 SunFire X4100 sliced into 4 Solaris Zones.
• 2 of the Zones are tagged to VLAN 4001 on e1000g2 (Intel NIC 2).
• 2 of the Zones are tagged to VLAN 4002 on e1000g3 (Intel NIC 3).
• 2 vSwitches…one for the web servers on VLAN 4001, and one for the mail servers on VLAN 4002.
• 2 VIPs on the shared vRouter which connects to the “Internet” and the two backend vSwitches.

• Its kind of strange and cool to design an entire web switching infrastructure virtually. If you've ever set up a complicated virtual net in VMware, it's similar…but oh so much more seductive.
• The two vSwitches are abso-positively separated. Despite having their VIPs on the same shared vRouter, neither the mail vSwitch nor the web vSwitch (or the real servers on them) can talk to each other. The wall between the vSwitches is steel-belted and brass riveted. That alone will keep my brain running with possibilities tonight!
• The SunFish WebUI, while it blows the doors off Alteon's WebUI and Java client (AlteonEMS) for configuring the web switch, needs some work to be as good as the AlteonEMS for day-to-day monitoring and ops. One feature I'd really like to see grafted into the SunFish CLI and WebUI is the Alteon's conception of “Apply”. Whenever any absent minded bloke (like myself) changes the configuration on an Alteon, those changes don't become active until you type apply. One of the cool side-effects of this paradigm is that you can darn near reconfigure an entire Alteon while its running, and have that baby instantly acquire all the facets of the new config when you type apply. Without apply you'd have to take the web switch completely out of battery to do a similarly drastic config change. It also means that in a mission-critical environment, you don't have to sweat bullets about typos taking down an SLB group or the switch with a fat fingered command.
• Also, the AlteonEMS has far superior heads-up displays of the health and status of all your load-balancing groups. You can find the displays in the SunFish WebUI, but they're scattered throughout the UI. There is no location you can see all of your groups, real servers and virtual servers on a single pane of glass like you can on the AlteonEMS. The AlteonEMS's single pane of glass also color-codes the individual elements (red, yellow or green) so you can instantly see whats dead and dying. If I'm wrong about this, PLEASE tell me. Its a feature we're going to miss.
• Unfortunately, the SunFish also inherits the Alteon's method of doing HA. We were really hoping to abandon VRRP and an SLB-hacked version of it on our web switches. For whatever reason, the SunFish does things the same way. While I'm sure there's some esoteric config out there pairing a web switch with a normal router using VRRP…is it really worth the hassle? I'd like to see a truly custom HA protocol where the two web switches merge as one…with stateful failover, and auto-sync of configuration between the units. Many firewalls now have this feature…lets bring it on over eh? VRRP is great for routers. End of story.

That's all the blabbering there is for the moment.Thus far, the SunFish is an incredible piece of engineering. I'm not quite ready to call it the JSF meets C17 of load-balancers, but I can hear the after burners warming up…

Tomorrow is dedicated to redundancy and SSL-offload! I'm stoked!

Technorati Tags: Sun, N1400V

Unsure what about Sun Cluster makes SXb41 crash uncontrollably, but the bug is not present in Solaris 10 6/06. It's truly a shame as mpathadm is an amazing tool  from build 41 I miss already. Mpathadm is by far the slickest, most well-designed storage multipathing utility I've had the privilege to use. If you don't need to run Sun Cluster I highly recommend running a current Solaris Express build. Mpathadm is worth it. You can fail MPXIO back and forth on a supported controller with the impunity of Superman
at a paper-mache convention.

Technorati Tags: Solaris Express, Sun Cluster