Big data seems to be all the rage these days. It's big, it's new, it's Hadoop-related, and it's typically in the public cloud. New startups and cloud offerings show up weekly, promising they'll finally get your data issues under control. They all promote the same idea: the migration to huge petabyte databases with almost "unlimited scalability" through the elasticity of the public cloud.
The reality is different than the hype lets on. As organizations try to consolidate their enterprise data into large databases that exist in the public cloud, they may be overlooking a few technical realities.
[ In the data center today, the action is in the private cloud. InfoWorld's experts take you through what you need to know to do it right in our "Private Cloud Deep Dive" PDF special report. | Also check out our "Cloud Security Deep Dive," our "Cloud Storage Deep Dive," and our "Cloud Services Deep Dive." ]
First, big data means big integration challenges. Thus, the ability to get the data from the enterprise to the public cloud may be problematic. Although you can certainly ship up a couple hundred thousand data records each day over the open Internet, in many cases we're talking millions of data records that must be transformed, translated, and synced from existing enterprise systems.
You bump your head on the bandwidth limitations quickly. Indeed, many enterprises are actually shipping USB drives via Federal Express to their public cloud providers to pump their big data system with current data.
Second, although security is indeed what you make of it in the cloud, it's typically cheaper to deal with data-level security in on-premise systems or in private clouds. In many cases, their security models and technology are less expensive. For example, you may have to encrypt data at rest in the public cloud -- but not in your old data center. Compliance is typically easier and cheaper when keeping data locally as well.
By the way, before you send in the hate mail, I'm not saying that big data is never a fit for public clouds, but you need to consider all the issues with the technologies. As with other architectural problems, you have to judge them on a case-by-case basis.
This article, "Big data and the cloud: A far from perfect fit," originally appeared at InfoWorld.com. Read more of David Linthicum's Cloud Computing blog and track the latest developments in cloud computing at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.