Data Wrangler Utilities

This tutorial covers the utility functions in datawrangler.util that help with data type detection, validation, and manipulation. These utilities are the building blocks that power data-wrangler’s automatic data type detection.

Overview

The datawrangler.util module provides essential helper functions:

• ``dataframe_like()``: Check if an object behaves like a DataFrame

• ``array_like()``: Detect array-like objects

• ``depth()``: Determine nesting depth of data structures

• ``btwn()``: Check if values fall within a range

These utilities are particularly useful when building custom data processing pipelines or extending data-wrangler’s functionality.

import datawrangler as dw
from datawrangler.util import dataframe_like, array_like, depth, btwn
import pandas as pd
import numpy as np
import os

Data Type Detection

Understanding how data-wrangler detects different data types is crucial for building robust data processing pipelines. Let’s explore the detection utilities:

# Test different data types with detection utilities
test_objects = [
    pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]}),  # True DataFrame
    {'A': [1, 2, 3], 'B': [4, 5, 6]},  # Dict (not DataFrame-like)
    np.array([[1, 2, 3], [4, 5, 6]]),  # NumPy array
    [[1, 2, 3], [4, 5, 6]],  # Nested list
    [1, 2, 3, 4, 5],  # Simple list
    "Hello World",  # String
    42  # Number
]

object_names = [
    "pandas DataFrame",
    "Dictionary",
    "NumPy Array",
    "Nested List",
    "Simple List",
    "String",
    "Number"
]

print("=== Data Type Detection Results ===")
print(f"{'Object Type':<20} {'DataFrame-like':<15} {'Array-like':<12} {'Depth':<8}")
print("-" * 60)

for obj, name in zip(test_objects, object_names):
    is_df_like = dataframe_like(obj)
    is_array_like = array_like(obj)
    obj_depth = depth(obj)

    print(f"{name:<20} {str(is_df_like):<15} {str(is_array_like):<12} {obj_depth:<8}")

=== Data Type Detection Results ===
Object Type          DataFrame-like  Array-like   Depth
------------------------------------------------------------
pandas DataFrame     True            True         1
Dictionary           False           False        0
NumPy Array          False           True         2
Nested List          False           True         2
Simple List          False           True         1
String               False           False        0
Number               False           True         0

Range checking with btwn

btwn(x, a, b) tests whether values fall within the inclusive range [a, b]. For an array it returns a single boolean that is True only when every element is in range – handy for validating that data lie in an expected interval.

values = np.array([0.2, 0.5, 0.9])
print(f'All of {values} in [0, 1]?   {btwn(values, 0, 1)}')
print(f'All of {values} in [0, 0.6]? {btwn(values, 0, 0.6)}')
print(f'Scalar 5 in [1, 10]?        {btwn(5, 1, 10)}')

# Element-wise membership is easy to build on top of the same idea
elementwise = [btwn(v, 0, 0.6) for v in values]
print(f'Element-wise in [0, 0.6]:   {elementwise}')

All of [0.2 0.5 0.9] in [0, 1]?   True
All of [0.2 0.5 0.9] in [0, 0.6]? False
Scalar 5 in [1, 10]?        True
Element-wise in [0, 0.6]:   [np.True_, np.True_, np.False_]

Reading tabular files with load_dataframe

load_dataframe powers data-wrangler’s file loading: it inspects a file’s extension and dispatches to the matching pandas reader, so the same call handles CSV, JSON, Excel, Parquet, and more with no extra configuration.

import tempfile
from datawrangler.io import load_dataframe

sample = pd.DataFrame({'city': ['NYC', 'SF', 'Chicago'], 'population_m': [8.5, 0.9, 2.7]})
tmp = tempfile.mkdtemp()

csv_path = os.path.join(tmp, 'cities.csv')
json_path = os.path.join(tmp, 'cities.json')
sample.to_csv(csv_path, index=False)
sample.to_json(json_path)

from_csv = load_dataframe(csv_path)    # extension '.csv' -> pandas.read_csv
from_json = load_dataframe(json_path)  # extension '.json' -> pandas.read_json
print(f'From CSV : {type(from_csv).__name__} of shape {from_csv.shape}')
print(f'From JSON: {type(from_json).__name__} of shape {from_json.shape}')
from_csv

From CSV : DataFrame of shape (3, 2)
From JSON: DataFrame of shape (3, 2)

	city	population_m
0	NYC	8.5
1	SF	0.9
2	Chicago	2.7