What is Apache Parquet?

  

Parquet is a columnar file format that is popular in the Hadoop ecosystem. It is similar to the RCFile format, but is more efficient in terms of storage and performance.

Parquet is designed to be efficient in terms of both storage and performance. It uses a columnar storage format, which means that data is stored in columns instead of rows. This makes it well-suited for analytical workloads, as data can be queried more efficiently. Parquet is also highly compressed, which helps to reduce storage costs.

One of the advantages of Parquet is that it supports complex data types, such as nested data structures. This makes it a good choice for data that is not easily represented in a tabular format. Parquet is also extensible, so new data types can be added in the future.

Parquet supports multiple compression codecs, including Snappy, GZip, and LZO. Parquet supports multiple encoding schemes, including dictionary encoding, run-length encoding, and frame-of-reference encoding. It also supports column-level encoding, which can further improve compression ratios.

Parquet is efficient for a variety of data processing workloads. Parquet supports several different data models, including nested data structures and schema evolution. Parquet also supports multiple languages, including Java, Scala, C# and Python.

Parquet is an open source project, and is available under the Apache License. You can find more information here.

Comments

Post a Comment

Popular posts from this blog

ZooKeeper as distributed consensus service

Image
The ability to bring consensus in a network of unreliable services is at the heart of any fault-tolerant distributed system. Apache ZooKeeper is a high-performance open-source server that provides a highly reliable distributed co-ordination and meets the production demands of the web-scale, mission critical applications. ZooKeeper is popular among multi-host, multi-process applications (C, Java bindings) running in data centers. This paper discusses how ZooKeeper is used to implement distributed consensus among the available services.   Every complex distributed application needs a service for maintaining configuration information, naming, providing distributed synchronization, and/or providing group services. Because of the complexity in implementing these solutions for distributed computing, the more natural preference is to use a tried, tested and reliable solution available in the market. ZooKeeper provides a simple interface to a centralized coordinating service. Th...
Read more

Recommendation systems

Image
Internet is flooded with information every day. Easy access to large amount of information along with difficulty in judging the validity of so much content can lead to information overload . As a result, e-commerce applications and social media sites are increasingly challenged to attract new users and retain existing ones. Based on user interests and preferences, these systems recommend items that may be of interest or value to the customers/users. Recommendation systems help users deal with the information-overload by giving them recommendations of products, etc.They help businesses make more profits, i.e., selling more products. Recommendation systems  are a type of information filtering system that uses the preferences of a group of people to make recommendations to other people. Some of the well-known features of recommendation system are: product recommendation for online shopping, social matching, targeted content/advertising etc. 35% of...
Read more

What is Apache Druid?

Image
  Apache D ru id is a high performance column - oriented , distributed data store designed for OL AP workload s . It is open source , supports horizontal scal ability , and has a low latency SQL interface . Apache D ru id was designed to address the needs of OL AP workload s , which are characterized by a few very large , slowly changing dimensions , and a large number of facts . Druid supports a wide variety of data sources , including relational databases , No SQL data stores , and flat files . Apache D ru id is highly scalable , supporting billions of rows and millions of events per second . Apache Druid has a low latency SQL interface that supports real - time queries on live data . Druid also has a powerful data ingestion engine that supports batch and streaming data sources . Apache D ru id has a rich SQL interface that supports most SQL operations , ...
Read more