Data Wrangler I/O Operations

This tutorial covers the I/O capabilities of data-wrangler, including loading and saving data from various sources and formats.

Overview

The datawrangler.io module provides seamless loading and saving of data from:

• Local files: CSV, JSON, text files, images, and more

• URLs: Load data directly from web sources

• Multiple formats: Automatic format detection based on file extensions

• Mixed sources: Handle lists of files/URLs with different formats

Let’s explore these capabilities with practical examples.

import datawrangler as dw
from datawrangler.io import load, save
import pandas as pd
import numpy as np
import os
from pathlib import Path

Loading Different File Formats

Data-wrangler automatically detects file formats and loads them appropriately. Let’s demonstrate with different file types:

# Create sample data files for demonstration
import tempfile

# Create a temporary directory for our examples
temp_dir = tempfile.mkdtemp()
print(f"Working in temporary directory: {temp_dir}")

# Create sample CSV file
csv_data = pd.DataFrame({
    'product': ['laptop', 'mouse', 'keyboard', 'monitor'],
    'price': [999.99, 25.50, 75.00, 300.00],
    'category': ['electronics', 'accessories', 'accessories', 'electronics']
})

csv_file = os.path.join(temp_dir, 'products.csv')
csv_data.to_csv(csv_file, index=False)

# Create sample text file
text_content = """Data science is transforming industries worldwide.
Machine learning enables computers to learn from data.
Natural language processing helps computers understand human language.
Data visualization makes complex data insights accessible."""

text_file = os.path.join(temp_dir, 'sample_text.txt')
with open(text_file, 'w') as f:
    f.write(text_content)

# Create sample JSON file
json_data = {
    'users': [
        {'name': 'Alice', 'age': 30, 'city': 'New York'},
        {'name': 'Bob', 'age': 25, 'city': 'San Francisco'},
        {'name': 'Charlie', 'age': 35, 'city': 'Chicago'}
    ]
}

json_file = os.path.join(temp_dir, 'users.json')
import json
with open(json_file, 'w') as f:
    json.dump(json_data, f)

print(f"Created files:")
print(f"- CSV: {csv_file}")
print(f"- Text: {text_file}")
print(f"- JSON: {json_file}")

Working in temporary directory: /var/folders/tp/qtzc39jx5w556wl5w3dj21wr0000gn/T/tmplgesr916
Created files:
- CSV: /var/folders/tp/qtzc39jx5w556wl5w3dj21wr0000gn/T/tmplgesr916/products.csv
- Text: /var/folders/tp/qtzc39jx5w556wl5w3dj21wr0000gn/T/tmplgesr916/sample_text.txt
- JSON: /var/folders/tp/qtzc39jx5w556wl5w3dj21wr0000gn/T/tmplgesr916/users.json

Loading local files

With the files created above, load detects each format from its extension and returns the natural Python object: CSV becomes a DataFrame, .txt becomes a string, and JSON becomes a DataFrame.

loaded_csv = load(csv_file)     # -> pandas DataFrame
loaded_text = load(text_file)   # -> str
loaded_json = load(json_file)   # -> pandas DataFrame

print(f'CSV  -> {type(loaded_csv).__name__} of shape {loaded_csv.shape}')
print(f'Text -> {type(loaded_text).__name__} ({len(loaded_text)} characters)')
print(f'JSON -> {type(loaded_json).__name__} of shape {loaded_json.shape}')
loaded_csv

CSV  -> DataFrame of shape (4, 3)
Text -> str (235 characters)
JSON -> DataFrame of shape (3, 1)

	product	price	category
0	laptop	999.99	electronics
1	mouse	25.50	accessories
2	keyboard	75.00	accessories
3	monitor	300.00	electronics

Loading remote files

load accepts URLs as well as local paths. Remote files are downloaded on first use and cached to disk, so loading the same URL again is fast and works offline.

url = 'https://raw.githubusercontent.com/ContextLab/data-wrangler/main/tests/resources/testdata.csv'
remote_df = load(url, index_col=0)
print(f'Loaded a {type(remote_df).__name__} of shape {remote_df.shape} from the web')
remote_df.head()

Loaded a DataFrame of shape (7, 5) from the web

	FirstDim	SecondDim	ThirdDim	FourthDim	FifthDim
ByTwos
0	1	2	3	4	5
2	2	4	6	8	10
4	3	6	9	12	15
5	4	8	12	16	20
6	5	10	15	20	25

Saving data

save(key, obj) writes data to disk. Strings are written as text and bytes are written directly; NumPy arrays use dtype='numpy' and arbitrary Python objects use dtype='pickle'. save stores the data at a cache location derived from the key; get_local_fname tells you where that is so you can read it back.

from datawrangler.io.io import get_local_fname

# Save a text note
save('analysis_notes.txt', 'data-wrangler makes preprocessing easy.')
print('Reloaded text:', load(get_local_fname('analysis_notes.txt')))

# Save a NumPy array
save('demo_array.npz', np.arange(12).reshape(3, 4), dtype='numpy')
restored = load(get_local_fname('demo_array.npz'))['arr_0']
print(f'Reloaded array of shape {restored.shape}')

Reloaded text: data-wrangler makes preprocessing easy.
Reloaded array of shape (3, 4)