The anti-spam industry sure does like their jello. At least that's what it seems like. Just try and compare “catch rates”…I dare you. The meaning keeps moving. Trying to wrap your mind around the cock-a-mamey way some vendor from East Timor calculates their accuracy is like trying to nail jello to the wall. The damn thing keeps moving. It's really not that hard folks.
You've basically got three questions:
* Of the spams that went through my engine what % of them were correctly identified?
* Of the innocents that went through my engine what % of them were correctly identified?
* Of ALL the messages that my engine processed what % where correctly classified (spam or innocent)?
That's it. That's all. Not that tough. Let's be honest…that's all an average Joe really wants to know.
Well…maybe that's not ALL an average Joe wants to know. He also wants to know you're being straight with him. Wanna know a dirty little secret….sshhh…be very quiet…..the term “catch-rate” only takes off accuracy points for spam misclassifications (i.e. false-negatives only). You want false-positives in that number? “Ha! We can't tell you that!” cackles the anti-spam vendor.
We call the three questions above Spam Classification Accuracy (SCA), Innocent Classification Accuracy (ICA) and Overall Classification Accuracy (OCA). Next time you're talking to an anti-spam vendor ask for them.** Push hard. At least you'll be able to make a clean choice.
**Just a heads up. Your friendly anti-spam vendor of choice will almost certainly stare at you like a wildebeest caught in the headlights if you ask for SCA, ICA and OCA by name. You'll probably have to tell them what you want in English…and speak reaaaalllyyy sloooow. They're not used to all the honesty.
Deep Dish:Way2tite – Situation 2wo:Global Underground 025: Toronto [UK] Disc 2[23:59]
Leave a reply