SQOOP - RDBMS to Hadoop
•
1 like•891 views
This document discusses using Sqoop to transfer data between relational databases and Hadoop. It begins by providing context on big data and Hadoop. It then introduces Sqoop as a tool for efficiently importing and exporting large amounts of structured data between databases and Hadoop. The document explains that Sqoop allows importing data from databases into HDFS for analysis and exporting summarized data back to databases. It also outlines how Sqoop works, including providing a pluggable connector mechanism and allowing scheduling of jobs.
Report
Share
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to SQOOP - RDBMS to Hadoop
Similar to SQOOP - RDBMS to Hadoop (20)
More from Sofian Hadiwijaya
More from Sofian Hadiwijaya (20)
Recently uploaded
Recently uploaded (20)
SQOOP - RDBMS to Hadoop
- 3. “Big data is like teenage sex: everyone talks about it, nobody really knows how to do it, everyone thinks everyone else is doing it, so everyone claims they are doing it…" - Dan Ariely -
- 4. Most of Startup in Indonesia using RDBMS But if we talk about BigData, everyone will talk about Hadoop
- 5. “In pioneer days they used oxen for heavy pulling, and when one ox couldn’t budge a log, they didn’t try to grow a larger ox. We shouldn’t be trying for bigger computers, but for more systems of computers" - Grace Hopper -
- 7. RDBMS focuses on relations, structures data in databases while the new Hadoop system processes unstructured data in parallelism on large clusters of inexpensive servers. Hadoop’s parallelism offers advantages in fast and credible results at low cost
- 9. • Apache Sqoop is a tool designed for efficiently transferring bulk data between Apache Hadoop and structured datastores such as relational databases. • Sqoop imports data from external structured datastores into HDFS or related systems like Hive and HBase. • Sqoop can also be used to export data from Hadoop and export it to external structured datastores such as relational databases and enterprise data warehouses. • Sqoop works with relational databases such as: Teradata, Netezza, Oracle, MySQL, Postgres, and HSQLDB. SQOOP What is SQOOP?
- 10. • As more organizations deploy Hadoop to analyse vast streams of information, they may find they need to transfer large amount of data between Hadoop and their existing databases, data warehouses and other data sources • Loading bulk data into Hadoop from production systems or accessing it from map-reduce applications running on a large cluster is a challenging task since transferring data using scripts is a inefficient and time-consuming task. SQOOP Why is SQOOP?
- 11. • Hadoop is great for storing massive data in terms of volume using HDFS • It Provides a scalable processing environment for structured and unstructured data • But it’s Batch-Oriented and thus not suitable for low latency interactive query operations • Sqoop is basically an ETL Tool used to copy data between HDFS and SQL databases • Import SQL data to HDFS for archival or analysis • Export HDFS to SQL ( e.g : summarized data used in a DW fact table ) SQOOP Hadoop-Sqoop?
- 12. Designed to efficiently transfer bulk data between Apache Hadoop and structured datastores such as relational databases, Apache Sqoop: • Allows data imports from external datastores and enterprise data warehouses into Hadoop • Parallelizes data transfer for fast performance and optimal system utilization • Copies data quickly from external systems to Hadoop • Makes data analysis more efficient • Mitigates excessive loads to external systems. SQOOP What Sqoop Does?
- 13. • Sqoop provides a pluggable connector mechanism for optimal connectivity to external systems • The Sqoop extension API provides a convenient framework for building new connectors which can be dropped into Sqoop installations to provide connectivity to various systems. • Sqoop itself comes bundled with various connectors that can be used for popular database and data warehousing systems. SQOOP How SQOOP Works?
- 17. SQOOP JOB • sqoop job —list • sqoop job --show myjob • sqoop job --exec myjob
- 18. SQOOP JOB • sqoop job —list • sqoop job --show myjob • sqoop job --exec myjob