Tutorial¶

This tutorial teaches you how to use rypipe to read files into Arrow tables and DataFrames, step by step. Each page builds on on the previous one, but every page is self-contained: you can jump straight to the topic you need and copy-paste the examples.

What is rypipe?¶

rypipe is a format-agnostic columnar ingestion framework. It reads row-oriented files (XML, CSV, JSONL, logs, etc.) and produces Apache Arrow tables with near-zero Python overhead.

The framework handles everything except format-specific parsing: parallel scheduling, memory-bounded execution, schema discovery, type coercion, filtering, and Arrow export. Adapters handle the parsing. Each adapter provides a Source class and re-exports rypipe's pipeline API so users need only one import.

your file
    |
    v
adapter (parsing)
    |
    v
rypipe engine
    |
    v
Arrow / pandas / Polars

This tutorial uses crxml, the published adapter for Crystal Reports XML exports. Every other adapter follows the same API; only the Source class name changes.

Installation¶

pip install "crxml[all]"

crxml pulls in the rypipe engine automatically; pyarrow, pandas, and polars are optional extras bundled in [all]. If you only need Arrow output without pandas or Polars:

pip install "crxml[pyarrow]"

A tiny taste¶

Download the sample file report.xml and save it next to your script. It is a small sales report with 15 rows.

from crxml import CrystalXMLSource, RenameFields, CastTypes, FilterRows, to_pandas

source = CrystalXMLSource("report.xml", row_tag="Details")

df = to_pandas(
    source
    | RenameFields({"Name": "name"})
    | CastTypes({"Amount": float})
    | FilterRows(field="Status", op="==", value="Active")
)

print(df)

Output:

$ python taste.py
             name   Department    Amount  Status        Date
0  Alice Johnson        Sales   15000.5  Active  2026-01-05
1      Bob Smith  Engineering    8500.0  Active  2026-01-06
2    Carol White    Marketing  12000.75  Active  2026-01-07
...

A few lines of code. The engine handled parsing, schema discovery, type coercion, filtering, and DataFrame export automatically.

How the tutorial works¶

Each page covers one topic:

1. First Steps, read a file and get a DataFrame.
2. Pipeline, chain transformations with |.
3. Stages, the transformation building blocks.
4. Sinks, get your data out.
5. Streaming, process files larger than RAM.
6. Configuration, all the options.

Want to read your own format?¶

If your files are not Crystal Reports XML, you can teach rypipe your format by writing a small adapter package. That is covered in the separate Building an Adapter track: you build a working adapter from scratch, no prior Rust experience required.

Recap¶

• rypipe is the ingestion engine; adapters add format-specific parsing.
• Install the adapter you need (for this tutorial: pip install crxml).
• A Source reads a file; | chains stages; a sink materializes the result.

Next: First Steps, read report.xml and get your first DataFrame.