What is Apache Hadoop?

 


Apache Hadoop is a distributed computing platform written in Java. It incorporates features similar to those of the Google File System and of MapReduce. Hadoop is an Apache project and is released under the Apache License. MapReduce is a programming model for large-scale data processing. It is designed to process and generate large data sets with a parallel, distributed algorithm on a cluster. In a MapReduce program, the map function is applied to each input record to produce a set of intermediate key/value pairs, and the reduce function is then applied to all the intermediate values associated with the same key to produce a set of output values. Apache Hadoop's MapReduce programming model takes advantage of data locality - the fact that data is often stored and processed near where it is used - to improve performance.

Apache Hadoop's HDFS is a distributed file system that stores files in a manner that provides high availability and reliability. HDFS
 stores data on commodity machines, providing very high aggregate bandwidth across the cluster. HDFS is designed to work with MapReduce, and it provides a number of features that improve the performance of MapReduce programs.

Apache Hadoop also includes a number of other components, such as a resource manager (Yarn) , a columnar storage system (HBase), and a machine learning library.

Apache Hadoop is often used in conjunction with Apache Spark, a fast, general-purpose cluster computing system. Hadoop can be deployed on a single node or on a cluster of thousands of nodes. It is fault-tolerant, scalable, and easy to maintain.

Apache Hadoop is written in Java and is available under the Apache License.


Comments

Post a Comment

Popular posts from this blog

ZooKeeper as distributed consensus service

Image
The ability to bring consensus in a network of unreliable services is at the heart of any fault-tolerant distributed system. Apache ZooKeeper is a high-performance open-source server that provides a highly reliable distributed co-ordination and meets the production demands of the web-scale, mission critical applications. ZooKeeper is popular among multi-host, multi-process applications (C, Java bindings) running in data centers. This paper discusses how ZooKeeper is used to implement distributed consensus among the available services.   Every complex distributed application needs a service for maintaining configuration information, naming, providing distributed synchronization, and/or providing group services. Because of the complexity in implementing these solutions for distributed computing, the more natural preference is to use a tried, tested and reliable solution available in the market. ZooKeeper provides a simple interface to a centralized coordinating service. Th...
Read more

Recommendation systems

Image
Internet is flooded with information every day. Easy access to large amount of information along with difficulty in judging the validity of so much content can lead to information overload . As a result, e-commerce applications and social media sites are increasingly challenged to attract new users and retain existing ones. Based on user interests and preferences, these systems recommend items that may be of interest or value to the customers/users. Recommendation systems help users deal with the information-overload by giving them recommendations of products, etc.They help businesses make more profits, i.e., selling more products. Recommendation systems  are a type of information filtering system that uses the preferences of a group of people to make recommendations to other people. Some of the well-known features of recommendation system are: product recommendation for online shopping, social matching, targeted content/advertising etc. 35% of...
Read more

What is Apache Druid?

Image
  Apache D ru id is a high performance column - oriented , distributed data store designed for OL AP workload s . It is open source , supports horizontal scal ability , and has a low latency SQL interface . Apache D ru id was designed to address the needs of OL AP workload s , which are characterized by a few very large , slowly changing dimensions , and a large number of facts . Druid supports a wide variety of data sources , including relational databases , No SQL data stores , and flat files . Apache D ru id is highly scalable , supporting billions of rows and millions of events per second . Apache Druid has a low latency SQL interface that supports real - time queries on live data . Druid also has a powerful data ingestion engine that supports batch and streaming data sources . Apache D ru id has a rich SQL interface that supports most SQL operations , ...
Read more