Big Data
What is big data?
Big data is a field that treats ways to analyze, systematically extract information from, or otherwise deal with data sets that are too large or complex to be dealt with by traditional data-processing application software. 1
Why is this important for hybrid cloud developers?
Data is getting larger, more complex, and moving faster than ever - a growth that is largely spurred by the widespread adoption of cloud computing.
With so many computers around the world connecting and transmitting data via the cloud, the challenge for developers is how to harness enormous amounts of data in a meaningful way.
Solution Sketch
- Batch data processing with Spark
- Real-time data processing with Kafka
Limitations
- TBD
Key open source projects
Cloud comparision
| Component | IBM Cloud | GCP | AWS | Azure | Operator |
|---|---|---|---|---|---|
| Managed Spark | Analytics Engine | Dataproc | EMR | HDInsight | Spark Operator |
| Stream Processing | Event Streams | Pub/Sub | MSK | Event Hubs | Kafka Operator |