How do I merge multiple PARQUET files?

Upload your PARQUET files using the file picker or drag and drop. You can add as many files as needed. Click "Merge" to combine them into a single file, then download the result.

Is there a limit on how many PARQUET files I can merge?

No, there's no limit on the number of files. You can merge hundreds of PARQUET files at once. All processing happens in the cloud with no file size restrictions.

Will the merged PARQUET file preserve my data structure?

Yes, the merge operation preserves all columns and data types. Files are combined row-by-row while maintaining the original structure.

Do I need to create an account to merge PARQUET files?

No account required for basic merging. Simply upload your files and download the merged result. Creating a free account unlocks additional features.

Is my data secure when merging files?

Yes, files are encrypted in transit and automatically deleted after processing. Your data is never stored permanently or accessed for any purpose other than merging.

Merge Parquet files online

Merge multiple Parquet files into a single file. Combine data from different sources quickly and easily.

Format

Files

Click anywhere to select filesor drag and drop files here

Accepts Parquet files (.parquet)

Trusted by over 60,000 every month

Parquet Merge Features

Multiple File Merging

Combine any number of files into a single consolidated file

Lightning-Fast Performance

Merge even large files instantly with optimized processing

Streamlined File Merging

Efficiently combine multiple files into a unified dataset

SQL-Powered Merging

Use SQL queries for advanced merging with custom join conditions

AI-Powered Assistance

Describe your merging needs in plain English for complex scenarios

Export Merged Data

Save your combined results in various formats

How to merge Parquet files in Python

Here are three effective ways to merge multiple Parquet files in Python using different libraries. Each approach has its own advantages depending on your specific needs and file sizes.

Merging Parquet files with Pandas

Pandas provides a straightforward approach for merging files and works well for most common data tasks:

First, let's install pandas if you haven't already:

bash

pip install pandas

Now we can load your parquet files into dataframes:

python

import pandas as pd

Let's load your first file:

python

df1 = pd.read_parquet('path/to/file1.parquet')

And your second file:

python

df2 = pd.read_parquet('path/to/file2.parquet')

Great! Now we can merge the dataframes using the concat function:

python

merged_df = pd.concat([df1, df2], ignore_index=True)

Finally, let's save your newly merged data to a file:

python

merged_df.to_parquet('path/to/merged_file.parquet', index=False)

Need to merge more than two files? No problem! Just add them to the list in the concat function:

python

merged_df = pd.concat([df1, df2, df3, df4], ignore_index=True)

Merging Parquet files with DuckDB

DuckDB is an in-process SQL OLAP database that's perfect for larger files and analytical workloads:

Let's start by installing DuckDB for Python:

bash

pip install duckdb

Now we'll import the library and create a connection:

python

import duckdb
con = duckdb.connect()

Here's a simple DuckDB query that will merge your parquet files using UNION ALL:

python

query = """
COPY (
  SELECT * FROM 'path/to/file1.parquet'
  UNION ALL
  SELECT * FROM 'path/to/file2.parquet'
) TO 'path/to/merged_file.parquet' (FORMAT 'parquet')
"""

Just run this query to perform the merge:

python

con.execute(query)

Got more than two files? Simply add more UNION ALL statements like this:

python

query = """
COPY (
  SELECT * FROM 'path/to/file1.parquet'
  UNION ALL
  SELECT * FROM 'path/to/file2.parquet'
  UNION ALL
  SELECT * FROM 'path/to/file3.parquet'
) TO 'path/to/merged_file.parquet' (FORMAT 'parquet')
"""

What's great about DuckDB is that it's incredibly efficient for large files - it processes data in a columnar format and can handle files that don't fit in memory. Perfect for those bigger merging jobs!

Merging Parquet files with ClickHouse

ClickHouse is a high-performance column-oriented database system that's excellent for large-scale data processing:

Let's begin by installing the ClickHouse Connect library for Python:

bash

pip install clickhouse-connect

Now we'll import the library and create a client connection:

python

import clickhouse_connect
client = clickhouse_connect.get_client(host='localhost')

Here's how you can merge your files using a single UNION ALL query:

python

query = f"""
CREATE TABLE merged_file AS
  SELECT * FROM file('path/to/file1.parquet', parquet)
  UNION ALL
  SELECT * FROM file('path/to/file2.parquet', parquet)
"""
client.command(query)

Then export your merged data to a file:

python

export_query = f"""
SELECT * FROM merged_file 
INTO OUTFILE 'path/to/merged_file.parquet'
FORMAT parquet
"""
client.command(export_query)

Need to merge more than two files? Just add more UNION ALL statements like this:

python

query = f"""
CREATE TABLE merged_file AS
  SELECT * FROM file('path/to/file1.parquet', parquet)
  UNION ALL
  SELECT * FROM file('path/to/file2.parquet', parquet)
  UNION ALL
  SELECT * FROM file('path/to/file3.parquet', parquet)
"""
client.command(query)

Want to skip the intermediate table? You can merge directly to a file in one step:

python

direct_query = f"""
SELECT * FROM file('path/to/file1.parquet', parquet)
UNION ALL
SELECT * FROM file('path/to/file2.parquet', parquet)
INTO OUTFILE 'path/to/merged_file.parquet'
FORMAT parquet
"""
client.command(direct_query)

ClickHouse really shines when you're working with massive datasets - it's a powerful columnar database that processes large volumes of data lightning fast, making it perfect for merging even the largest files.

Frequently Asked Questions

More Parquet Tools

Parquet View

View and analyze Parquet files online

Parquet Filter

Filter Parquet data by column values

Parquet Sort

Sort Parquet files by any column

Parquet Sample

Download sample Parquet files

Parquet Generate

Generate test Parquet data

Parquet to CSV

Convert Parquet files to CSV

Parquet to JSON

Convert Parquet files to JSON

Parquet to TSV

Convert Parquet files to TSV

Parquet to Excel

Convert Parquet files to Excel