Data Management Glossary
Apache Parquet
What is Apache Parquet?
Apache Parquet is an open-source, column-oriented file format for storing structured and semi-structured data. Instead of writing data row by row, the way a spreadsheet or a CSV file does, Parquet groups and compresses the values in each column together. That layout lets query engines read only the columns a query actually needs, skip the rest, and decompress far less data to get an answer.
Parquet is built for structured data, think of relational tables such as sales transactions or sensor readings, and semi-structured data, such as nested, JSON-like records with repeated or optional fields. It is not built to store raw unstructured content such as documents, images, video, audio, or genomic sequencing files directly. That distinction matters for anyone trying to bring file and object data into a modern data lakehouse, and it is where Komprise fits in.
Apache Parquet is one of the most widely adopted file formats in the data lake and lakehouse ecosystem. Table formats including Apache Iceberg, Delta Lake, and Apache Hudi are built as a metadata and transaction layer on top of Parquet files, adding schema evolution, time travel, and ACID transactions to the columnar storage Parquet provides underneath.
Source: Apache Parquet Documentation, Apache Software Foundation
An Apache Software Foundation Project
Parquet started as a joint effort between engineers at Twitter and Cloudera, using the record-shredding and assembly algorithm described in Google’s Dremel paper. The first release, Parquet 1.0, shipped in July 2013. On April 27, 2015, Apache Parquet became a top-level Apache Software Foundation project, governed the same way as Apache Hadoop, Apache Spark, Apache Hive, and Apache Iceberg.
Apache governance is part of why Parquet became the default rather than staying a Twitter/Cloudera-specific format: it is vendor-neutral, its specification is public, and virtually every major query engine and cloud data warehouse (Spark, Hive, Presto/Trino, Snowflake, Databricks, Amazon Athena, BigQuery) implements native Parquet support.
Source: The Apache Software Foundation Blog, “The Apache Software Foundation Announces Apache Parquet as a Top-Level Project” (April 27, 2015)
Apache Parquet vs. CSV
CSV is a row-based, plain-text format. Every field, in every row, is stored and read as text, in the order it appears in the file. Parquet is a binary, column-oriented format. Values for a single column are stored together, compressed using an encoding suited to that column’s data type, and indexed with metadata that lets a query engine skip entire columns and row groups it does not need.
The practical differences:
- Storage size. Parquet’s per-column compression typically produces much smaller files than the same data in CSV. In DuckDB’s TPC-H benchmark at scale factor 20, Parquet files were 5 times smaller than the equivalent CSV files (3.2 GB vs. 16 GB).
Source: DuckDB, “CSV Files: Dethroning Parquet as the Ultimate Storage File Format, or Not?” (December 5, 2024) - Query cost and speed. Because a query engine reads only the columns a query touches instead of scanning every row and column, Parquet queries scan less data and run faster. In the same DuckDB TPC-H benchmark, analytical queries ran 7 to 10 times faster on Parquet than CSV, with a complex join completing in 2 seconds on Parquet versus 20 seconds on CSV. AWS reports that converting data to a columnar format such as Parquet cuts per-query costs by 30% to 90% on Amazon Athena.
Sources: DuckDB, “CSV Files: Dethroning Parquet as the Ultimate Storage File Format, or Not?“; AWS Big Data Blog, “Top 10 Performance Tuning Tips for Amazon Athena“
- Schema and types. CSV has no built-in schema. Every value is a string until something downstream imposes types on it. Parquet embeds a defined schema, with native types such as integer, timestamp, and nested struct, directly in the file.
- Row-level edits. CSV is simple to append to or edit line by line. Parquet is written in immutable column chunks, so row-level inserts, updates, and deletes require rewriting affected chunks, which is exactly why table formats such as Apache Iceberg exist on top of Parquet.
- When to use each. CSV still works well for small files, quick data exchange, and human-readable inspection. Parquet is the better choice once data needs to be queried repeatedly, joined with other tables, or stored at scale, which is why it has become the default file format for data lakes and lakehouses.
Quick reference:
| Attribute | CSV | Apache Parquet |
|---|---|---|
| Structure | Row-based, plain text | Column-based, binary |
| Schema | None, all values are strings | Embedded, typed schema |
| Compression | Minimal | High, per-column encoding |
| Typical file size vs. CSV | Baseline | About 5x smaller, per DuckDB’s TPC-H benchmark |
| Query speed vs. CSV | Baseline | About 7 to 10x faster, per DuckDB’s TPC-H benchmark |
| Row-level edits | Simple, line by line | Requires rewriting column chunks |
| Best for | Small files, quick exchange, human review | Repeated queries, joins, data at scale |
Why Structured and Semi-Structured Data Sources Matter Here
Parquet’s compression and query performance gains come from a precondition: the data already has rows and columns, or at minimum a describable, semi-structured record shape. A sales table, a clickstream log, a JSON API payload, all of these map cleanly to Parquet’s column-oriented layout.
Enterprise unstructured data does not start out that way. A file share full of engineering drawings, medical images, video assets, or genomic sequencing output has no rows or columns until someone or something describes it that way, file name, size, owner, creation date, content type, embedded tags, classification labels, extracted entities. Until that description exists, none of it can be represented as a Parquet column, and it cannot be queried alongside structured data in Snowflake, Databricks, or any other lakehouse engine.
Komprise COO and cofounder Krishna Subramanian put it directly: current data ingestion techniques such as ETL and ELT “are built for structured and semi-structured (e.g. JSON, CSV) data,” because they focus on copying raw data without extracting a schema first. Unstructured file and object data does not fit that mold, and forcing it through the same pipeline means moving petabytes of raw files before anyone can query them.
The Case for Unstructured Data Management
Building that structured description by hand does not scale. IDC and Gartner estimate that 80% to 90% of enterprise data is unstructured, growing roughly three times faster than structured data, and Komprise notes that 99% of it remains dark to AI simply because there is no easy way to query it. It typically lives across dozens of NAS systems, object stores, and cloud accounts, with no single index and no consistent naming or tagging convention across silos.
Point solutions that scan one storage system at a time, or ETL pipelines that copy raw files to a destination before they can be analyzed, break down at petabyte scale: the copy is slow, expensive, and stale the moment it finishes.
Getting file and object data into a Parquet-backed lakehouse table requires solving this at the metadata layer, indexing every file and object across every silo, enriching that index with content-derived metadata, and keeping it current, without moving the underlying files first.
The Komprise Role: Transparent File Tables
Komprise Transparent File Tables, a capability within Komprise Intelligent Data Management, closes the gap between raw unstructured files and Parquet-backed lakehouse tables.
Komprise indexes file and object data across NAS, object storage, and cloud into the Global Metadatabase, capturing system metadata (file type, size, age, owner, access patterns) alongside content-derived metadata extracted through KAPPA data services and Smart Data Workflows, including header extraction and sensitive data tagging. A data engineer then uses Deep Analytics to query the Global Metadatabase and define the exact subset of files to expose.
That curated result exports as an Apache Iceberg table, complete with a defined schema and a dynamic pointer to each underlying file through patented Transparent Move Technology. No raw files move at export time. Data engineers query the result in Snowflake, Databricks, or any other Iceberg-compatible engine using standard SQL, joining unstructured file metadata with structured data from ERP, EHR, or CRM systems in the same query. If an AI pipeline needs the actual file content, Komprise fetches it on demand from wherever it already lives.
From there, Parquet enters the picture as the delivery format for downstream AI work. In one published Transparent File Tables use case, a machine learning engineer at a healthcare provider queries a Transparent File Table enriched with AI-generated tags extracted from DICOM files, joins it with structured patient cohort data, and exports the curated result as Parquet for ingestion into a RAG pipeline or fine-tuning workflow. In terms of the Komprise AI Data Platform’s five layers, storage, metadata and discovery, classification and governance, enrichment and curation, and AI delivery, Transparent File Tables sits at the handoff between the enrichment and curation layer, where Komprise builds the enriched schema, and the AI delivery layer, where Parquet-backed formats such as Iceberg tables and exported Parquet files are what lakehouses, RAG pipelines, and fine-tuning workflows actually consume.
Komprise vs. Manual Approaches: Getting Unstructured Data Into Parquet-Backed Tables
| Evaluation Criteria | Without Komprise | With Komprise |
|---|---|---|
| Unstructured Data Sources | Manual, per-silo Custom scripts to flatten files from each NAS or object store before any Parquet conversion |
NAS & Cloud Indexed automatically at exabyte scale across every silo |
| Path to a Parquet-Backed Table | Custom ETL Raw files copied and reformatted by hand before they can be represented as columns |
Automated Deep Analytics exports curated metadata directly as an Apache Iceberg table via Transparent File Tables |
| Data Movement | Heavy Full copy of raw files to the lakehouse or warehouse before conversion |
Zero Only schema and metadata are exported; raw files stay in place |
| Schema | None Designed and maintained by hand for every source |
Rich Schema Komprise-enriched from content scanning and metadata extraction |
| Data Quality | Poor Raw, inconsistent file and object metadata with duplicates and gaps |
High Curated, classified, enriched metadata ready to load into Parquet columns |
| Sensitive Data Handling | None No standard way to screen files before they enter a shared table |
Automated Built-in PII and regex detection before export |
| Governance | Manual Fragmented access controls across tools and silos |
Automated Access permissions and audit trails carried into every downstream Parquet-based table |
| Time to Insight | Slow Weeks to months to transfer, reformat, and validate data |
Immediate Query file and object metadata where it lives |
Apache Parquet FAQs
What is Apache Parquet?
Apache Parquet is an open-source, column-oriented file format for storing structured and semi-structured data efficiently. It compresses data by column, embeds a typed schema in the file, and lets query engines read only the columns a query needs, which is why it has become the standard storage format for data lakes and lakehouses.
What is the difference between Parquet and CSV?
CSV stores data as plain text, row by row, with no schema and no compression beyond what you apply externally. Parquet stores data as compressed, typed columns with an embedded schema. In practical terms, Parquet files are typically several times smaller than the same data in CSV (DuckDB’s TPC-H benchmark found Parquet files 5 times smaller), and queries against Parquet run significantly faster, since the query engine reads only the relevant columns instead of scanning every row and field in the file.
Who created Apache Parquet, and is it an Apache Software Foundation project?
Parquet was created jointly by engineers at Twitter and Cloudera, drawing on the record-shredding and assembly algorithm from Google’s Dremel paper. The first release shipped in July 2013, and Apache Parquet became a top-level Apache Software Foundation project on April 27, 2015, the same governance model as Apache Hadoop and Apache Spark.
Is Apache Parquet used for unstructured data?
Not directly. Parquet is designed for structured data (tables) and semi-structured data (nested records with a describable shape), not for raw unstructured content such as documents, images, or video files. To bring unstructured file and object data into a Parquet-backed lakehouse table, the file and object metadata has to be extracted, enriched, and structured first, which is the specific problem Komprise Transparent File Tables is designed to solve.
How does Komprise use Apache Parquet to make unstructured data queryable?
Komprise does not convert raw files into Parquet directly. Komprise Transparent File Tables indexes and enriches unstructured file and object metadata in the Global Metadatabase, then exports curated subsets as Apache Iceberg tables. Since Iceberg tables are backed by Parquet files, the enriched metadata becomes queryable in Snowflake, Databricks, and other Parquet-native engines using standard SQL, with no raw file movement and no per-source ETL pipeline required.
Do I need to move my files to use Parquet-based analytics on unstructured data?
No. Komprise Transparent File Tables exports only the metadata schema as an Iceberg table backed by Parquet. The raw files stay in place across NAS, object, and cloud storage. If an AI or analytics pipeline needs the actual file content, Komprise retrieves it on demand through Transparent Move Technology.
Which layer of the Komprise AI Data Platform does Apache Parquet support?
The AI Data Platform spans five layers: storage and data sources, metadata and discovery, classification and governance, enrichment and curation, and AI delivery and consumption. Komprise operates at layers 2 through 4, indexing, governing, and enriching unstructured data. Parquet-based delivery, whether through a Transparent File Tables Iceberg export or a Parquet file exported for a RAG or fine-tuning pipeline, sits at the handoff into layer 5, AI delivery and consumption, where lakehouses, analytics platforms, and AI pipelines consume the curated result.
AI Data Platform
Apache Iceberg
Data Catalog
Data Fabric
Data Lake
Data Lakehouse
ETL
Global Metadatabase
Metadata Catalog
Transparent File Tables
Unstructured Data Metadata Management