In this tutorial, we immerse ourselves in ModinA powerful reference replacement for pandas that operates parallel computers to considerably speed up data workflows. By important modin.pandas as a PD, we transform our Pandas code into a distributed computing power. Our goal here is to understand how Modin works through real world data operations, such as Groupby, joints, cleaning and analysis of chronological series, while operating on Google Colar. We have each task compared to the standard Pandas library to see how fast and more economical modin in memory can be.
!pip install "modin(ray)" -q
import warnings
warnings.filterwarnings('ignore')
import numpy as np
import pandas as pd
import time
import os
from typing import Dict, Any
import modin.pandas as mpd
import ray
ray.init(ignore_reinit_error=True, num_cpus=2)
print(f"Ray initialized with {ray.cluster_resources()}")We start by installing Modin with the Backend Ray, which allows transparent parallelized pandas operations in Google Colar. We delete unnecessary warnings to keep the output clean and clear. Then, we import all the necessary libraries and initialize the shelves with 2 processors, by preparing our environment for the distributed processing of data data.
def benchmark_operation(pandas_func, modin_func, data, operation_name: str) -> Dict(str, Any):
"""Compare pandas vs modin performance"""
start_time = time.time()
pandas_result = pandas_func(data('pandas'))
pandas_time = time.time() - start_time
start_time = time.time()
modin_result = modin_func(data('modin'))
modin_time = time.time() - start_time
speedup = pandas_time / modin_time if modin_time > 0 else float('inf')
print(f"\n{operation_name}:")
print(f" Pandas: {pandas_time:.3f}s")
print(f" Modin: {modin_time:.3f}s")
print(f" Speedup: {speedup:.2f}x")
return {
'operation': operation_name,
'pandas_time': pandas_time,
'modin_time': modin_time,
'speedup': speedup
}
We define a benchmark_operation function to compare the execution time of a specific task using both Pandas and Modin. By performing each operation and recording its duration, we calculate the Speedup Modin offers. This provides us with a clear and measurable way to assess the performance gains for each operation we test.
def create_large_dataset(rows: int = 1_000_000):
"""Generate synthetic dataset for testing"""
np.random.seed(42)
data = {
'customer_id': np.random.randint(1, 50000, rows),
'transaction_amount': np.random.exponential(50, rows),
'category': np.random.choice(('Electronics', 'Clothing', 'Food', 'Books', 'Sports'), rows),
'region': np.random.choice(('North', 'South', 'East', 'West'), rows),
'date': pd.date_range('2020-01-01', periods=rows, freq='H'),
'is_weekend': np.random.choice((True, False), rows, p=(0.3, 0.7)),
'rating': np.random.uniform(1, 5, rows),
'quantity': np.random.poisson(3, rows) + 1,
'discount_rate': np.random.beta(2, 5, rows),
'age_group': np.random.choice(('18-25', '26-35', '36-45', '46-55', '55+'), rows)
}
pandas_df = pd.DataFrame(data)
modin_df = mpd.DataFrame(data)
print(f"Dataset created: {rows:,} rows × {len(data)} columns")
print(f"Memory usage: {pandas_df.memory_usage(deep=True).sum() / 1024**2:.1f} MB")
return {'pandas': pandas_df, 'modin': modin_df}
dataset = create_large_dataset(500_000)
print("\n" + "="*60)
print("ADVANCED MODIN OPERATIONS BENCHMARK")
print("="*60)We define a CREATE_LARGE_DATASET function to generate a set of rich synthetic data with 500,000 lines that imitate real transactional data, including customer information, purchasing models and horodatages. We create both pandas and modin versions of this data set so that we can compare them side by side. After generating the data, we display its dimensions and its memory footprint, by preparing the terrain for advanced Modin operations.
def complex_groupby(df):
return df.groupby(('category', 'region')).agg({
'transaction_amount': ('sum', 'mean', 'std', 'count'),
'rating': ('mean', 'min', 'max'),
'quantity': 'sum'
}).round(2)
groupby_results = benchmark_operation(
complex_groupby, complex_groupby, dataset, "Complex GroupBy Aggregation"
)
We define a Complex_groupby function to carry out group operations on several levels on the data set by grouping it by category and region. We then aggregate several columns using functions such as the sum, the average, the standard deviation and the counting. Finally, we have this operation on Pandas and Modin to measure how much Modin performs the aggregations of heavy groups more quickly.
def advanced_cleaning(df):
df_clean = df.copy()
Q1 = df_clean('transaction_amount').quantile(0.25)
Q3 = df_clean('transaction_amount').quantile(0.75)
IQR = Q3 - Q1
df_clean = df_clean(
(df_clean('transaction_amount') >= Q1 - 1.5 * IQR) &
(df_clean('transaction_amount') <= Q3 + 1.5 * IQR)
)
df_clean('transaction_score') = (
df_clean('transaction_amount') * df_clean('rating') * df_clean('quantity')
)
df_clean('is_high_value') = df_clean('transaction_amount') > df_clean('transaction_amount').median()
return df_clean
cleaning_results = benchmark_operation(
advanced_cleaning, advanced_cleaning, dataset, "Advanced Data Cleaning"
)
We define the Advanced_Cleaning function to simulate a pipeline for precontaining the real world. First of all, we delete aberrant values using the IQR method to ensure cleaner information. Then, we carry out the engineering of the functionalities by creating a new metric called transaction_score and by labeling high value transactions. Finally, we have this cleaning logic using pandas and modin to see how they manage complex transformations on large data sets.
def time_series_analysis(df):
df_ts = df.copy()
df_ts = df_ts.set_index('date')
daily_sum = df_ts.groupby(df_ts.index.date)('transaction_amount').sum()
daily_mean = df_ts.groupby(df_ts.index.date)('transaction_amount').mean()
daily_count = df_ts.groupby(df_ts.index.date)('transaction_amount').count()
daily_rating = df_ts.groupby(df_ts.index.date)('rating').mean()
daily_stats = type(df)({
'transaction_sum': daily_sum,
'transaction_mean': daily_mean,
'transaction_count': daily_count,
'rating_mean': daily_rating
})
daily_stats('rolling_mean_7d') = daily_stats('transaction_sum').rolling(window=7).mean()
return daily_stats
ts_results = benchmark_operation(
time_series_analysis, time_series_analysis, dataset, "Time Series Analysis"
)
We define the Time_Series_Analysis function to explore daily trends by re -chanting transaction data over time. We define the date column as an indication, calculate daily aggregations such as the sum, the average, the number and the average note, and compilies them in a new dataaframa. To capture models in the longer term, we also add an average of 7 days. Finally, we count this pipeline of chronological series with pandas and modin to compare their effectiveness on temporal data.
def create_lookup_data():
"""Create lookup tables for joins"""
categories_data = {
'category': ('Electronics', 'Clothing', 'Food', 'Books', 'Sports'),
'commission_rate': (0.15, 0.20, 0.10, 0.12, 0.18),
'target_audience': ('Tech Enthusiasts', 'Fashion Forward', 'Food Lovers', 'Readers', 'Athletes')
}
regions_data = {
'region': ('North', 'South', 'East', 'West'),
'tax_rate': (0.08, 0.06, 0.09, 0.07),
'shipping_cost': (5.99, 4.99, 6.99, 5.49)
}
return {
'pandas': {
'categories': pd.DataFrame(categories_data),
'regions': pd.DataFrame(regions_data)
},
'modin': {
'categories': mpd.DataFrame(categories_data),
'regions': mpd.DataFrame(regions_data)
}
}
lookup_data = create_lookup_data()
We define the Create_Lookup_Data function to generate two reference tables: one for product categories and another for regions, each containing relevant metadata such as commission rates, tax rates and shipping costs. We prepare these research tables in Pandas and Modin formats so that we can use them later in joint operations and compare their performance in the two libraries.
def advanced_joins(df, lookup):
result = df.merge(lookup('categories'), on='category', how='left')
result = result.merge(lookup('regions'), on='region', how='left')
result('commission_amount') = result('transaction_amount') * result('commission_rate')
result('tax_amount') = result('transaction_amount') * result('tax_rate')
result('total_cost') = result('transaction_amount') + result('tax_amount') + result('shipping_cost')
return result
join_results = benchmark_operation(
lambda df: advanced_joins(df, lookup_data('pandas')),
lambda df: advanced_joins(df, lookup_data('modin')),
dataset,
"Advanced Joins & Calculations"
)
We define the Advanced_joins function to enrich our main data set by merging it with category and region research tables. After having done the joints, we calculate additional fields, such as Commission_amount, Tax_amount and Total_cost, to simulate the financial calculations of the real world. Finally, we have all this pipeline of joint and calculation using both Pandas and Modin to assess the way Modin manages complex operations in several stages.
print("\n" + "="*60)
print("MEMORY EFFICIENCY COMPARISON")
print("="*60)
def get_memory_usage(df, name):
"""Get memory usage of dataframe"""
if hasattr(df, '_to_pandas'):
memory_mb = df.memory_usage(deep=True).sum() / 1024**2
else:
memory_mb = df.memory_usage(deep=True).sum() / 1024**2
print(f"{name} memory usage: {memory_mb:.1f} MB")
return memory_mb
pandas_memory = get_memory_usage(dataset('pandas'), "Pandas")
modin_memory = get_memory_usage(dataset('modin'), "Modin")We are now moving to the development towards the use of memory and print a section header to highlight this comparison. In the Get_memory_usage function, we calculate the imprint of the memory of pandas and modin dataframes using their internal methods Memory_usage. We guarantee compatibility with modin by checking the _TO_Pandas attribute. This helps us to assess how much Modin effectively manages memory compared to pandas, in particular with large data sets.
print("\n" + "="*60)
print("PERFORMANCE SUMMARY")
print("="*60)
results = (groupby_results, cleaning_results, ts_results, join_results)
avg_speedup = sum(r('speedup') for r in results) / len(results)
print(f"\nAverage Speedup: {avg_speedup:.2f}x")
print(f"Best Operation: {max(results, key=lambda x: x('speedup'))('operation')} "
f"({max(results, key=lambda x: x('speedup'))('speedup'):.2f}x)")
print("\nDetailed Results:")
for result in results:
print(f" {result('operation')}: {result('speedup'):.2f}x speedup")
print("\n" + "="*60)
print("MODIN BEST PRACTICES")
print("="*60)
best_practices = (
"1. Use 'import modin.pandas as pd' to replace pandas completely",
"2. Modin works best with operations on large datasets (>100MB)",
"3. Ray backend is most stable; Dask for distributed clusters",
"4. Some pandas functions may fall back to pandas automatically",
"5. Use .to_pandas() to convert Modin DataFrame to pandas when needed",
"6. Profile your specific workload - speedup varies by operation type",
"7. Modin excels at: groupby, join, apply, and large data I/O operations"
)
for tip in best_practices:
print(tip)
ray.shutdown()
print("\n✅ Tutorial completed successfully!")
print("🚀 Modin is now ready to scale your pandas workflows!")We conclude our tutorial by summarizing performance references in all the operations tested, calculating the average acceleration that Modin has obtained on pandas. We also highlight the most efficient operation, offering a clear view of the place where Modin excels the most. Then, we share a set of best practices to effectively use Modin, including advice on compatibility, performance profiling and conversion between pandas and modin. Finally, we closed Ray.
In conclusion, we saw first -hand how Modin can overcome our workflows pandas with a minimum of modifications to our code. Whether they are complex aggregations, analysis of chronological series or joints with high memory intensity, Modin offers evolutionary performance for daily tasks, in particular on platforms like Google Colab. With the power of Ray under the hood and the API Pandas almost complete compatibility, Modin makes effortlessly to operate with more important data sets.
Discover the Codes. All the merit of this research goes to researchers in this project. Also, don't hesitate to follow us TwitterAnd YouTube And don't forget to join our Subseubdredit 100k + ml and subscribe to Our newsletter.
Nikhil is an intern consultant at Marktechpost. It pursues a double degree integrated into materials at the Indian Kharagpur Institute of Technology. Nikhil is an IA / ML enthusiast who is still looking for applications in fields like biomaterials and biomedical sciences. With a strong experience in material science, he explores new progress and creates opportunities to contribute.
