Configuration¶

This page collects the options you will use day to day: the Source constructor, streaming, and error handling. The constructor options below are the ones crxml accepts; other adapters may differ, so check your adapter's documentation.

Source constructor options¶

from crxml import CrystalXMLSource

src = CrystalXMLSource(
    "report.xml",
    row_tag="Details",          # str: which XML element is one row
    engine="auto",              # "auto", "stream", "columnar", "parallel"
    threads=None,               # int | None: adapter default; use os.cpu_count() for all cores
    memory=None,                # str | int: bound memory, e.g. "512MB"
    field_mapping=None,         # dict[str, str]: rename columns
    drop_fields=None,           # list[str]: skip columns entirely
    filter=None,                # dict: pushdown filter spec
    field_types=None,           # dict[str, str]: type hints, e.g. {"Amount": "float64"}
    dictionary_columns=None,    # list[str]: dictionary-encode columns
    schema=None,                # list[str]: project exactly these columns, in this order
    auto_dict=False,            # bool: auto dictionary-encode low-cardinality strings
    use_mmap=True,              # bool: memory-mapped file I/O
    batch_size=1024,            # int: Python-side batch size for iteration/export
)

You only need row_tag to get started. The rest are optimizations and conveniences:

# Typed, filtered, renamed: applied during parsing, not after
src = CrystalXMLSource(
    "report.xml",
    row_tag="Details",
    field_types={"Amount": "float64"},
    filter={"field": "Status", "op": "==", "value": "Active"},
)
print(src.to_arrow().num_rows)  # 12

The filter dict uses the same spec as FilterRows. See the FilterRows reference for all forms and operators.

Schema projection¶

schema=[...] is a projection + order declaration: the output contains exactly the listed columns, in the listed order. Fields in the data that are not listed are skipped during parsing (never decoded or materialized; this is the performance win), listed columns that are absent from some or all rows come out null-filled, and extra/unknown fields in the data are ignored rather than raising an error. field_mapping renames apply before schema matching (list the renamed names), field_types still applies to schema columns, and a column listed in both schema and drop_fields is dropped (drop_fields wins).

src = CrystalXMLSource("report.xml", row_tag="Details", schema=["Name", "Amount"])
print(src.schema())  # ['Name', 'Amount']: Department/Status/Date never parsed

Schema discovery¶

discover_schema() scans a file once and returns its column names, which is handy for exploring an unknown export:

from crxml import discover_schema

print(discover_schema("report.xml", row_tag="Details"))
# ['Name', 'Department', 'Amount', 'Status', 'Date']

It accepts the same field_mapping, drop_fields, filter, field_types, dictionary_columns, schema, and auto_dict options as the Source constructor.

Streaming options¶

iter_record_batches() (on Sources and pipelines) takes:

total = 0
for batch in src.iter_record_batches(memory="64MB", threads=16):
    total += batch.num_rows

See Streaming for the full patterns.

How crxml plugs in¶

You never wire anything up yourself: import crxml registers the adapter with the engine under the name crxml for .xml files, and every CrystalXMLSource uses it from there. Registration only matters if you are writing your own adapter package. See Building an Adapter.

Exceptions¶

Catch ValueError for usage errors and XmlError for data problems:

from crxml import CrystalXMLSource, XmlError

try:
    table = CrystalXMLSource("report.xml", row_tag="Details").to_arrow()
except XmlError as e:
    print("Could not parse:", e)

Recap¶

• The Source constructor is the full API; row_tag is the only required option.
• Constructor options apply renames, drops, filters, and types during parsing.
• discover_schema() inspects an unknown file without a full read.
• Streaming is configured with memory= and threads= on iter_record_batches().
• import crxml registers the adapter; there is nothing to wire up.

Next: Building an Adapter, teach the engine your own format.