[ ]:
Data Wrangler Decorators Part 2: Advanced Decorators
This tutorial covers advanced decorator functionality in data-wrangler, including interpolation, stacking/unstacking operations, and building complex data processing pipelines.
Advanced Decorators Overview
Beyond the basic @funnel decorator, data-wrangler provides specialized decorators for:
``@interpolate``: Automatic handling of missing data
``@apply_stacked``: Operations on stacked (melted) data
``@apply_unstacked``: Operations on unstacked (pivoted) data
Custom decorator combinations: Chaining decorators for complex workflows
These decorators enable sophisticated data preprocessing pipelines with minimal code.
[1]:
import datawrangler as dw
import pandas as pd
import numpy as np
from datawrangler.decorate import funnel, interpolate, apply_stacked, apply_unstacked
import matplotlib.pyplot as plt
The @interpolate Decorator
The @interpolate decorator automatically handles missing data by applying interpolation methods before passing data to your function. This is particularly useful for time series analysis and data cleaning pipelines.
[2]:
# Create sample data with missing values
np.random.seed(42)
dates = pd.date_range('2024-01-01', periods=20, freq='D')
values = np.random.randn(20).cumsum()
# Introduce some missing values
values[5:8] = np.nan
values[15] = np.nan
# Create DataFrame with missing data
timeseries_data = pd.DataFrame({
'date': dates,
'value': values,
'category': ['A'] * 10 + ['B'] * 10
})
print("Original data with missing values:")
print(timeseries_data)
print(f"\nMissing values: {timeseries_data['value'].isna().sum()}")
Original data with missing values:
date value category
0 2024-01-01 0.496714 A
1 2024-01-02 0.358450 A
2 2024-01-03 1.006138 A
3 2024-01-04 2.529168 A
4 2024-01-05 2.295015 A
5 2024-01-06 NaN A
6 2024-01-07 NaN A
7 2024-01-08 NaN A
8 2024-01-09 3.938051 A
9 2024-01-10 4.480611 A
10 2024-01-11 4.017193 B
11 2024-01-12 3.551464 B
12 2024-01-13 3.793426 B
13 2024-01-14 1.880146 B
14 2024-01-15 0.155228 B
15 2024-01-16 NaN B
16 2024-01-17 -1.419891 B
17 2024-01-18 -1.105643 B
18 2024-01-19 -2.013668 B
19 2024-01-20 -3.425971 B
Missing values: 4
[3]:
# Define a function that computes rolling statistics
@funnel
@interpolate
def compute_rolling_stats(data, window=5):
"""Compute rolling statistics on clean data"""
if 'value' not in data.columns:
return pd.DataFrame()
result = pd.DataFrame({
'rolling_mean': data['value'].rolling(window=window).mean(),
'rolling_std': data['value'].rolling(window=window).std(),
'rolling_min': data['value'].rolling(window=window).min(),
'rolling_max': data['value'].rolling(window=window).max()
})
return result
# Apply to data with missing values - interpolation happens automatically
rolling_stats = compute_rolling_stats(timeseries_data, interp_kwargs={'method': 'linear'})
print("Rolling statistics computed on interpolated data:")
print(rolling_stats.head(10))
# Verify no missing values in the processed data
print(f"\nMissing values after interpolation: {rolling_stats.isna().sum().sum()}")
Rolling statistics computed on interpolated data:
rolling_mean rolling_std rolling_min rolling_max
0 NaN NaN NaN NaN
1 NaN NaN NaN NaN
2 NaN NaN NaN NaN
3 NaN NaN NaN NaN
4 1.337097 1.013924 0.358450 2.529168
5 1.778909 1.037209 0.358450 2.705774
6 2.330526 0.798959 1.006138 3.116533
7 2.834756 0.489986 2.295015 3.527292
8 3.116533 0.649467 2.295015 3.938051
9 3.553652 0.692402 2.705774 4.480611
Missing values after interpolation: 16
/Users/jmanning/data-wrangler/datawrangler/decorate/decorate.py:340: FutureWarning: DataFrame.interpolate with object dtype is deprecated and will raise in a future version. Call obj.infer_objects(copy=False) before interpolating instead.
data = data.interpolate(**kwargs)
Stacking and unstacking
Many datasets are naturally a collection of DataFrames – one per subject, session, file, or trial – that all share the same columns. dw.stack concatenates such a list into a single DataFrame with a hierarchical (MultiIndex) row index that remembers which rows came from which original DataFrame. dw.unstack is the exact inverse, splitting a stacked DataFrame back into the original list.
Why is this useful? Stacking lets you run one vectorized operation over all the data at once (fitting a model, normalizing, computing statistics) instead of looping, while unstacking recovers the per-DataFrame structure so downstream code still sees the individual pieces.
[4]:
from datawrangler import stack, unstack
# Three DataFrames that share the same columns (e.g. one per experiment subject)
subjects = [pd.DataFrame(np.random.rand(4, 2), columns=['x', 'y']) for _ in range(3)]
stacked = stack(subjects)
print(f'Stacked shape: {stacked.shape} with a {stacked.index.nlevels}-level index')
print(stacked.head(6))
restored = unstack(stacked)
print(f'\nUnstacked back into {len(restored)} DataFrames of shape {restored[0].shape}')
Stacked shape: (12, 2) with a 2-level index
x y
ID
0 0 0.456070 0.785176
1 0.199674 0.514234
2 0.592415 0.046450
3 0.607545 0.170524
1 0 0.065052 0.948886
1 0.965632 0.808397
Unstacked back into 3 DataFrames of shape (4, 2)
@apply_stacked and @apply_unstacked
These decorators wire stacking/unstacking into a function automatically:
``@apply_stacked``: stacks a list of DataFrames, runs your function once on the combined data, then unstacks the result. Use it for operations that should treat all the data together (e.g. a global mean).
``@apply_unstacked``: the opposite – given a stacked DataFrame, it splits the data, applies your function to each piece independently, then re-stacks. Use it for per-group operations.
[5]:
from datawrangler.decorate import apply_stacked, apply_unstacked
@apply_stacked
def demean_globally(df):
"""Subtract the grand mean computed across ALL subjects at once."""
return df - df.mean()
@apply_unstacked
def zscore_each(df):
"""Standardize each subject independently."""
return (df - df.mean()) / df.std()
# apply_stacked takes the list, works on the pooled data, and returns a list
pooled = demean_globally(subjects)
print(f'@apply_stacked returned {len(pooled)} DataFrames; pooled mean is now ~0: '
f'{stack(pooled).mean().abs().max():.2e}')
# apply_unstacked takes the stacked frame, works per-subject, and returns a stacked frame
standardized = zscore_each(stacked)
print(f'@apply_unstacked returned a {type(standardized).__name__} of shape {standardized.shape}')
@apply_stacked returned 3 DataFrames; pooled mean is now ~0: 3.70e-17
@apply_unstacked returned a DataFrame of shape (12, 2)