Data Wrangler Decorators Part 1: The @funnel Decorator

This tutorial introduces the powerful @funnel decorator, which automatically converts function inputs to pandas DataFrames. This allows you to write functions that work seamlessly with any data type that data-wrangler supports.

The @funnel Decorator

The @funnel decorator is the cornerstone of data-wrangler’s function integration system. It automatically wrangles function arguments into DataFrames, allowing your functions to work with:

• Raw arrays, lists, and nested data structures

• Text data (automatically embedded using NLP models)

• Files and URLs

• Mixed data types

• Any other data type supported by data-wrangler

Let’s see how this works in practice.

import datawrangler as dw
import pandas as pd
import numpy as np
from datawrangler import funnel
import matplotlib.pyplot as plt

Basic Example: Numerical Analysis Function

Let’s start with a simple function that computes basic statistics. Without @funnel, this would only work with DataFrames:

# Define a function that works on DataFrames
@funnel
def compute_stats(data):
    """Compute basic statistics for numerical data"""
    return {
        'mean': data.mean().mean(),
        'std': data.std().mean(),
        'shape': data.shape,
        'columns': list(data.columns)
    }

# Test with different data types
print("=== Testing with different data types ===")

# 1. Raw numpy array
array_data = np.random.randn(10, 5)
print("\\n1. NumPy Array:")
print(f"Input shape: {array_data.shape}")
stats = compute_stats(array_data)
print(f"Result: {stats}")

# 2. 2D NumPy array (built from a nested Python list)
list_data = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
print("\\n2. 2D NumPy array:")
print(f"Input: {list_data}")
stats = compute_stats(list_data)
print(f"Result: {stats}")

# 3. Already a DataFrame
df_data = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})
print("\\n3. Pandas DataFrame:")
print(f"Input shape: {df_data.shape}")
stats = compute_stats(df_data)
print(f"Result: {stats}")

=== Testing with different data types ===
\n1. NumPy Array:
Input shape: (10, 5)
Result: {'mean': np.float64(-0.16432272636920753), 'std': np.float64(1.0205481350367525), 'shape': (10, 5), 'columns': [0, 1, 2, 3, 4]}
\n2. 2D NumPy array:
Input: [[1 2 3]
 [4 5 6]
 [7 8 9]]
Result: {'mean': np.float64(5.0), 'std': np.float64(3.0), 'shape': (3, 3), 'columns': [0, 1, 2]}
\n3. Pandas DataFrame:
Input shape: (3, 2)
Result: {'mean': np.float64(3.5), 'std': np.float64(1.0), 'shape': (3, 2), 'columns': ['A', 'B']}

Text Processing with @funnel

One of the most powerful features is how @funnel handles text data automatically. Let’s create a function that analyzes text sentiment and see how it works with different text inputs:

@funnel
def analyze_text_dimensions(text_data, text_kwargs={'model': 'all-MiniLM-L6-v2'}):
    """Analyze the dimensionality and characteristics of text embeddings"""
    print(f"Received DataFrame with shape: {text_data.shape}")
    print(f"Data type: {type(text_data)}")
    print(f"Columns: {list(text_data.columns)}")

    # Basic statistics about the embeddings
    stats = {
        'embedding_dimensions': text_data.shape[1],
        'num_texts': text_data.shape[0],
        'mean_embedding_magnitude': np.sqrt((text_data ** 2).sum(axis=1)).mean(),
        'embedding_std': text_data.std().mean()
    }

    return stats

# Test with different text inputs
print("=== Testing text processing with @funnel ===")

# 1. Single text string
single_text = "This is a sample sentence for analysis."
print("\\n1. Single text string:")
print(f"Input: '{single_text}'")
result = analyze_text_dimensions(single_text)
print(f"Result: {result}")

# 2. List of texts
text_list = [
    "Data science is fascinating.",
    "Machine learning transforms industries.",
    "Natural language processing enables AI communication.",
    "Data wrangling simplifies preprocessing."
]
print("\\n2. List of texts:")
print(f"Input: {len(text_list)} texts")
result = analyze_text_dimensions(text_list)
print(f"Result: {result}")

=== Testing text processing with @funnel ===
\n1. Single text string:
Input: 'This is a sample sentence for analysis.'

/Users/jmanning/.pyenv/versions/3.10.12/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm

loading corpus: minipedia...done!Received DataFrame with shape: (1, 50)
Data type: <class 'pandas.core.frame.DataFrame'>
Columns: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49]
Result: {'embedding_dimensions': 50, 'num_texts': 1, 'mean_embedding_magnitude': np.float64(0.5147815070493398), 'embedding_std': nan}
\n2. List of texts:
Input: 4 texts
Received DataFrame with shape: (4, 50)
Data type: <class 'pandas.core.frame.DataFrame'>
Columns: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49]
Result: {'embedding_dimensions': 50, 'num_texts': 4, 'mean_embedding_magnitude': np.float64(0.5669376195429721), 'embedding_std': np.float64(0.018058002593314436)}

The list_generalizer Decorator

@funnel converts inputs to DataFrames. A lower-level building block, @list_generalizer, does something complementary: it lets a function that operates on a single object automatically accept a list of objects, applying itself to each element and returning a list of results. data-wrangler uses it internally so that every wrangling function transparently supports lists of inputs.

from datawrangler.decorate import list_generalizer

@list_generalizer
def describe(x):
    """Summarize a single array-like object."""
    arr = np.asarray(x)
    return {'shape': arr.shape, 'mean': float(arr.mean())}

# A single object -> a single result
print('Single object:')
print(describe(np.arange(10)))

# A list of objects -> a list of results (no manual loop required)
print()
print('List of objects:')
for result in describe([np.arange(10), np.ones((3, 3)), [5, 5, 5]]):
    print(result)

Single object:
{'shape': (10,), 'mean': 4.5}

List of objects:
{'shape': (10,), 'mean': 4.5}
{'shape': (3, 3), 'mean': 1.0}
{'shape': (3,), 'mean': 5.0}