datawrangler package

Subpackages

• datawrangler.core
  • get_default_options()
  • apply_defaults()
  • update_dict()
  • set_dataframe_backend()
  • get_dataframe_backend()
  • reset_dataframe_backend()
• datawrangler.decorate
  • list_generalizer()
  • funnel()
  • interpolate()
  • apply_stacked()
  • apply_unstacked()
• datawrangler.io
  • load()
  • save()
• datawrangler.util
  • btwn()
  • depth()
  • array_like()
  • dataframe_like()
• datawrangler.zoo
  • datawrangler.zoo.array
    • is_array()
    • wrangle_array()
  • datawrangler.zoo.dataframe
    • is_dataframe()
    • is_multiindex_dataframe()
    • wrangle_dataframe()
  • datawrangler.zoo.text
    • is_text()
    • wrangle_text()
    • get_corpus()
    • get_text_model()
    • apply_text_model()
    • to_str_list()
    • get_text()
  • datawrangler.zoo.null
    • is_null()
    • wrangle_null()
  • datawrangler.zoo.polars_dataframe
    • is_polars_dataframe()
    • is_polars_lazyframe()
    • wrangle_polars_dataframe()
    • create_polars_dataframe()
    • pandas_to_polars()
    • polars_to_pandas()

Module contents

Data Wrangler: Transform messy data into clean DataFrames (pandas or Polars)

Data Wrangler is a Python package that automatically transforms various data types (arrays, text, files, URLs, etc.) into clean, consistent DataFrame format using either pandas or Polars backends. It specializes in text data processing using modern NLP models and offers 2-100x performance improvements with Polars.

Key Features: - Dual backend support: pandas (default) or Polars for high performance - Automatic data type detection and conversion - Text embedding using sentence-transformers and sklearn models - Function decorators for seamless DataFrame integration - Support for files, URLs, and mixed data types - Configurable processing pipeline

Basic Usage:

>>> import datawrangler as dw
>>> df = dw.wrangle(your_data)  # pandas DataFrame (default)
>>> df_fast = dw.wrangle(your_data, backend='polars')  # Polars DataFrame

# With text data using sentence-transformers >>> text_df = dw.wrangle([“Hello world”, “Another text”], … text_kwargs={‘model’: ‘all-MiniLM-L6-v2’}, … backend=’polars’) # 2-100x faster with Polars

# Using the @funnel decorator (backend-agnostic) >>> @dw.funnel … def your_function(df): … return df.mean()

Backend Differences:

• pandas:

  • Full feature compatibility with named indexes

  • Index names preserved during processing

  • Slower performance on large datasets

  • Extensive ecosystem support

• Polars:

  • High performance (2-100x faster on large datasets)

  • Position-based indexing only (no named indexes)

  • Index names not preserved during backend conversion

  • Limited interpolation support in decorators

  • Growing ecosystem, may have fewer integrations

Choose pandas for: Small datasets, complex index operations, maximum compatibility Choose Polars for: Large datasets, performance-critical applications, simple workflows

Requirements: - Python 3.9+ - Optional: Install with [hf] extras for sentence-transformers support

pip install “pydata-wrangler[hf]”

Version: 0.5.0+ (dual pandas/Polars backend; NumPy 2.0+ and pandas 2.0+ compatible)

datawrangler.funnel(f)

A decorator that coerces any data passed into the function into a DataFrame (pandas or Polars) or a list of DataFrames

Parameters

param f:

a function of the form f(data, *args, **kwargs) that assumes data is either a DataFrame or a list of DataFrames

Returns

return:

A decorated function that supports any wrangle-able data format. The decorated function accepts an optional ‘backend’ keyword argument (‘pandas’ or ‘polars’) to specify the DataFrame backend.

Notes

The decorated function can be called with: - backend=’pandas’: Convert inputs to pandas DataFrames (default) - backend=’polars’: Convert inputs to Polars DataFrames for better performance

datawrangler.stack(data, names=None, keys=None, verify_integrity=False, sort=False, copy=True, ignore_index=False, levels=None)

Take a list of DataFrames with the same number of columns and (optionally) a list of names (of the same length as the original list; default: range(len(x))). Return a single MultiIndex DataFrame where the original DataFrames are stacked vertically, with the data names as their level 1 indices and their original indices as their level 2 indices.

Parameters

param data:

A single DataFrame or a list of DataFrames with must matching columns.

param names:

names for the levels in the resulting hierarchical index. (Default: None)

param keys:

if multiple levels passed, should contain tuples. Construct hierarchical index using the passed keys as the outermost level.

param verify_integrity:

check whether the new concatenated axis contains duplicates. This can be very expensive relative to the actual data concatenation. (Default: False)

param sort:

sort non-concatenation axis if it is not already aligned when join is ‘outer’. This has no effect when join=’inner’, which already preserves the order of the non-concatenation axis. (Default: False)

param copy:

if False, do not copy data unnecessarily. (Default: True)

param ignore_index:

if True, do not use the index values along the concatenation axis. The resulting axis will be labeled 0, …, n - 1. This is useful if you are concatenating objects where the concatenation axis does not have meaningful indexing information. Note the index values on the other axes are still respected in the join. (Default: False)

param levels:

specific levels (unique values) to use for constructing a MultiIndex. Otherwise they will be inferred from the keys. (Default: None)

param kwargs:

any other keyword arguments will be passed to datawrangler.decorate.funnel

Returns

return:

a single MultiIndex DataFrame

datawrangler.unstack(x)

Turn a MultiIndex DataFrame into a list of DataFrames, using the unique top-level index values to divide the data. If the dataset is a “regular” (non-MultiIndex) DataFrame, it is returned as a list with a single element, containing the un-modified DataFrame

Parameters

param x:

a single DataFrame, a MultiIndex DataFrame, or a list of DataFrames

Returns

return:

a list of one or more DataFrames

datawrangler.wrangle(x, return_dtype=False, backend=None, **kwargs)

Turn messy data into clean DataFrames (pandas or Polars)

Automatically detects and converts various data types into consistent DataFrame format. Specializes in text processing using modern NLP models and handles mixed data types.

Parameters

param x:

data in any format. Supported datatypes: - Numpy Arrays, array-like objects, or paths to files that store array-like objects - DataFrames (pandas or Polars), dataframe-like objects, or paths to files that store dataframe-like objects - Polars LazyFrames - Text strings, lists of strings, or paths to plain text files - Mixed lists or nested lists of the above types

param return_dtype:

if True, also return the auto-detected datatype(s) of each dataset. Default: False

param backend:

str, optional The DataFrame backend to use (‘pandas’ or ‘polars’). If None, uses the default backend (pandas)

param kwargs:

control how data are wrangled:

• array_kwargs: passed to wrangle_array function to control how arrays are handled

• dataframe_kwargs: passed to wrangle_dataframe function to control how dataframes are handled

• text_kwargs: passed to wrangle_text function to control how text data are handled. Common text_kwargs options (simplified API):

  • {‘model’: ‘all-MiniLM-L6-v2’} for sentence-transformers

  • {‘model’: ‘CountVectorizer’} for sklearn text vectorization

  • {‘model’: [‘CountVectorizer’, ‘LatentDirichletAllocation’]} for sklearn pipeline

  Also supports full dict format for advanced configuration:

  • {‘model’: {‘model’: ‘all-MiniLM-L6-v2’, ‘args’: [], ‘kwargs’: {}}}

Any other keyword arguments are passed to all wrangle functions.

Returns

return:

a DataFrame (pandas or Polars), or a list of DataFrames, containing the wrangled data

Examples

>>> import datawrangler as dw
>>> # Convert array to pandas DataFrame (default)
>>> df = dw.wrangle([1, 2, 3])
>>> # Convert array to Polars DataFrame
>>> df_polars = dw.wrangle([1, 2, 3], backend='polars')
>>> # Process text with sentence-transformers model
>>> text_df = dw.wrangle(["Hello", "World"], text_kwargs={'model': 'all-MiniLM-L6-v2'})
>>> # Handle mixed data types and return detected types
>>> mixed_df, dtypes = dw.wrangle([df, text_df], return_dtype=True)