Comments for Jason's .plan

Comment on Rabbits and warrens. by Jason

Jason — Sat, 24 Jul 2010 02:26:30 +0000

Multiple channels to my experience really give you two things: concurrency and guaranteed order.

for example, let’s say you’re breaking a file into 100 messages and re-assembling them on the otherside. If you transmit all 100 on one channel you can be guaranteed that message 1 is consumed before message 2 and so on.

Having multiple channels…say one per thread let’s you publish multiple messages more in parallel. Also, let’s say you’ve got one thread that needs to wait on a reply queue after publishing, that channel will not be able to do publishing again until it’s exited consume mode. That would keep your other threads from being able to send or consume messages.

Comment on Cloud-scale DBs in the cloud…just a quickie by Jason

Jason — Sat, 24 Jul 2010 02:24:41 +0000

You’re right in your comments…my point is the hype is way overblown as to where that tipping point of outgrowing an RDBMS is. Also, even accounting for power the markup on the smaller servers is extremely high.

Comment on Rabbits and warrens. by Rachid

Rachid — Tue, 20 Jul 2010 12:50:17 +0000

Thank you for this excellent article

Comment on Rabbits and warrens. by Darius Damalakas

Darius Damalakas — Wed, 14 Jul 2010 13:20:25 +0000

Hackeron, the download links work fine, just tested, examples work fine. AMQP really rocks

Comment on Rabbits and warrens. by [RabbitMQ+Python入门经典] 兔子和兔子窝 | 有氧呼吸

[RabbitMQ+Python入门经典] 兔子和兔子窝 | 有氧呼吸 — Tue, 06 Jul 2010 06:49:44 +0000

[...] RabbitMQ作为一个工业级的消息队列服务器，在其客户端手册列表的Python段当中推荐了一篇blog，作为RabbitMQ+Python的入门手册再合适不过了。不过，正如其标题Rabbit and Warrens（兔子和养兔场）一样，这篇英文写的相当俏皮，以至于对于我等非英文读者来说不像一般的技术文档那么好懂，所以，翻译一下吧。翻译过了，希望其他人可以少用一些时间。翻译水平有限，不可能像原文一样俏皮，部分地方可能就意译了，希望以容易懂为准。想看看老外的幽默的，推荐去看原文，其实，也不是那么难理解…… [...]

Comment on Rabbits and warrens. by George

George — Fri, 25 Jun 2010 12:32:20 +0000

Thanks for this writeup! A year and a half later it’s still almost the only resource for AMQP from Python.

One thing it’s not clear for me from the example is what are (multiple) channels used for and how do they relate to exchanges and queues. In which occasions would a client or a server create more than one channels ? If a client does “msg1 = chan1.basic_get(‘po_box’)” and “msg2 = chan2.basic_get(‘po_box’)” back to back, is msg1==msg2 or they are two successive messages (my guess is the latter but then I don’t see the difference of using one vs two channels).

Comment on Cloud-scale DBs in the cloud…just a quickie by Alex Leverington

Alex Leverington — Sat, 19 Jun 2010 11:43:47 +0000

I believe your analysis is accurate and it makes sense: If you have beefy data processing needs, it’s more efficient to use beefy servers. In answer to your query about why “cheap” (in bulk) servers cost more, it’s because they actually cost more! I’ll explain…

For 28 commodity servers you’re looking at powering 2 racks (@110v) in a climate controlled room. At 3amps, the electric bill to cool and power the servers would be around $1300/mo. What consumes 3amps when a 100W processor is < 1 amp?

The power consumed by servers are for the CPU, Fans, and Disks. When you take all those disks and cram them into one machine, based on your example @28 servers, you’re eliminating 104 CPU cores, 26 CPU Fans, and probably 104 system fans — bringing power usage down to about $48/mo rather than $1300/mo.

If all you want to do is store 11TB of data, it’s not a matter of beefy server + rdbms or many small servers + cassandra. The benefit of those 28 servers + cassandra is that cassandra will not need more than a single core on each of those servers and you can probably use 2-3 of the other cores for distributing computations. If you don’t need that kind of process distribution, chances are you can do all your processing with an RDBMS and you’re better off with one beefy server. In most cases, if someone was spanning their 11TB of data across 28 servers, it’s because they can optimally split up their dataset into 14 segments.

In a nutshell, most folks who expand out from the “beefy database” infrastructure do so because their data processing needs exceed the CPU, memory, or network capacity of one server. In most cases, the limitation of a beefy server isn’t the amount of storage available but the latency of random seek access and ability of the system to quickly store+index new data.

Comment on Rabbits and warrens. by Cal Leeming

Cal Leeming — Thu, 27 May 2010 20:46:20 +0000

Heya,

Thank you ever so much for this introduction into RabbitMQ, it’s definitely been one of the more interesting documents I’ve read in quite a long time, and has been quite informative!

Cal

Comment on Cloud-scale DBs in the cloud…just a quickie by Jason

Jason — Thu, 18 Mar 2010 22:48:09 +0000

The question is, how do you get 11TB of highly available database. And as noted in the article, the servers used at the cloud provider were dedicated servers not cloud VMs per se. If you have a large dataset, something like Cassandra will cost you disproportionally more money than using MySQL vertically scaled and then partitioned (when deployed at a cloud provider). Alternately, you could vertically scale and use 3 beefy Cassandra nodes, but that obviates the point of the architecture.

My real goal with this posting was to get cloud providers to realize they’ve got to cut the costs on their 1U servers if things like Cassandra are going to be affordable when deployed in the cloud with multi-TB datasets. The 1U costs are way out of line with what the same cloud provider charges for the beefier box. If you bundle colo space/power/bandwidth into the costs of the 1U and 4U servers deployed on your own you’re still at a 1/5 ratio for 1Us vs 1/11 ratio for 4Us (ratio being calculated as (cost per month at cloud provider/cost of hardware itself plus colo/power/bandwidth). Ideally, 1U or 4U beefy the ratio would be the same.

Comment on Cloud-scale DBs in the cloud…just a quickie by Jonathan Ellis

Jonathan Ellis — Thu, 18 Mar 2010 16:12:32 +0000

You’re not getting apples to apples here. The amount of disk is similar but you have an order of magnitude more CPU and RAM in the scale-out hardware, which is huge for many workloads.

That said, I also blogged recently about why cloud VMs aren’t usually the right fit vs bare metal.