What is Big Data
Big data is an assortment of such huge and complex data that becomes very difficult to capture, store, process, retrieve and analyze it.
The 5 V’s properties of Big data
Velocity Speed at which data is emanating and changes are occurring between the diverse data sets
Volume This refers to the sheer volume of data being generated every second
Variety Can use structured as well as unstructured data
Veracity Data reliable and trust. Verifying and validating the data
Value Having access to big data is all well and good but that’s only useful if we can turn it into a value
Challenges in Big Data
- Dealing with data growth
- Analyzing in timely manner
- Integrating desperate data sources
- Validating Data
- Securing Data
What is Hadoop
Apache Hadoop is an open-source software platform for distributed storage and distributed processing of very large data sets. Clusters built from commodity hardware.
When to use Hadoop?
- Data Size and Data Diversity
- Multiple Frameworks for Big Data
- Lifetime Data Availability
- Batch
When NOT to use Hadoop?
- Real Time Analytics
- Not a Replacement for Existing Infrastructure
- Multiple Smaller Datasets
Hadoop Ecosystem
