Filter Parquet files online

How to filter Parquet files in Python

Here are three effective ways to filter Parquet files in Python using different libraries. Each approach has its own advantages depending on your specific needs and file sizes.

Filtering Parquet files with Pandas

Pandas provides a straightforward approach for filtering files and works well for most common data tasks:

First, we need to install pandas:

pip install pandas

Next, import pandas and load your parquet file:

import pandas as pd
df = pd.read_parquet('path/to/file.parquet')

Apply a filter to your data. For example, to filter rows where a column named 'value' is greater than 100:

filtered_df = df[df['value'] > 100]

You can also apply multiple conditions using logical operators:

# Filter rows where value > 100 AND category is 'A'
filtered_df = df[(df['value'] > 100) & (df['category'] == 'A')]
# Filter rows where value > 100 OR category is 'A'
filtered_df = df[(df['value'] > 100) | (df['category'] == 'A')]

Finally, save the filtered data back to a file:

filtered_df.to_parquet('filtered_file.parquet', index=False)

Filtering Parquet files with DuckDB

DuckDB is an in-process SQL OLAP database that's perfect for larger files and analytical workloads:

First, we need to install duckdb:

pip install duckdb

Import DuckDB and set up your environment:

import duckdb

Use SQL to filter your data directly from the file. For example, to filter rows where a column named 'value' is greater than 100:

query = f"SELECT * FROM 'path/to/file.parquet' WHERE value > 100"
result = duckdb.sql(query).fetchdf()

You can use any SQL WHERE clause for more complex filtering:

# Filter with multiple conditions
query = f"SELECT * FROM 'path/to/file.parquet' WHERE value > 100 AND category = 'A'"
result = duckdb.sql(query).fetchdf()

Save the filtered data to a new file:

duckdb.sql(f"COPY result TO 'filtered_file.parquet'").fetchdf()

Filtering Parquet files with ClickHouse

ClickHouse is a high-performance column-oriented database system that's excellent for large-scale data processing:

First, we need to install clickhouse-connect:

pip install clickhouse-connect

Import ClickHouse and set up a client connection:

import clickhouse_connect
client = clickhouse_connect.get_client(host='localhost', port=8123)

Use SQL to filter your data. For example, to filter rows where a column named 'value' is greater than 100:

query = f"SELECT * FROM file('path/to/file.parquet') WHERE value > 100"
result = client.query(query).result_set

ClickHouse supports complex SQL filtering with multiple conditions:

# Filter with multiple conditions and aggregation
query = f"SELECT category, AVG(value) as avg_value FROM file('path/to/file.parquet') WHERE value > 100 GROUP BY category"
result = client.query(query).result_set

Save the filtered data to a new file:

client.command(f"SELECT * FROM file('path/to/file.parquet') WHERE value > 100 INTO OUTFILE 'filtered_file.parquet'")