Skip to content

Changelog

All notable changes to rypipe, newest first. Versions follow semantic versioning and are tagged in the repository.

[Unreleased]

[0.6.1] - 2026-10-15

Changed

  • Every unsafe block is documented, and undocumented ones fail the build. #![deny(clippy::undocumented_unsafe_blocks)] is enabled in rypipe-core, and the SIMD mask helpers and their callers, the counting allocator, the memory map, and the post-validation UTF-8 borrow each carry a SAFETY (or # Safety) note describing its invariant.

Notes

  • Cargo.lock is now tracked. CI and downstream audits resolve the exact dependency set instead of re-resolving on every build; Dependabot keeps it current.

CI

  • Core tests run under overflow checks. A release-mode cargo test -p rypipe-core with -C overflow-checks=on fails on integer overflow that the normal release profile would wrap silently.

[0.6.0] - 2026-10-14

Added

  • ColumnarSink::should_stop. A default-false hook a parser can poll between records to abort the current parse early. Bounded single-pass sinks set it when the consumer has gone away, so a dropped streaming iterator stops the worker promptly instead of finishing the whole input.

Changed

  • Bounded streaming is now a single pass. BoundedExecutor's run_stream / run_bytes_stream (and their _with_stats / consumer variants) parse the whole input through one TableBuilder and spill finished RecordBatches to the consumer via the new SpillSink, replacing the previous "parse each pre-split chunk into its own builder, then merge::extend every chunk" path. This removes a second full copy of all column data, the per-chunk engine churn, and the split-planning pass from the single-threaded streaming path. On the xmlstreamer CDATA feed this moved single-thread bounded streaming from ~0.6x to ~1.0-1.1x of a raw quick-xml scan; flat feeds from ~1.0x to ~1.1-1.5x. Delivered rows, columns, schema-union behaviour, StreamStats, budget accounting and the memory-budget contract are unchanged.
  • Mapped streaming parses in place. run_mapped_stream reads the memory-mapped input directly instead of dropping the mapping and re-reading each chunk through a reusable buffer.
  • Oversize-regime batching preserved. When the budget cannot hold a single row, the spilling sink keeps the MIN_ROWS_OVERSIZE_BATCH floor and does not flush early on capacity, so per-row batches cannot explode consumer cost.
  • Multi-delimiter BlockMasks and quote-aware next_unquoted. Every delimiter's bitmask is computed from one 64-byte load, and next_unquoted derives an inside-quotes mask with a SWAR prefix-XOR, so delimiters inside quoted values are skipped with no per-byte state machine.
  • ColumnarSink::put_str_at string fast path. Push a known &str into a known slot without the Value round trip; TableBuilder writes straight into the column, falling back when a strict memory budget or a filter row buffer is active.

Documentation

  • Ecosystem overview page. A new integrations page lists adapters and links them from the documentation navigation.

Notes

  • BoundedExecutor::plan_chunks is retained as a planning utility (and for its split-cap test) but is no longer on the streaming hot path. TableBuilder::bytes_used is now unused internally.

0.5.1 - 2026-10-01

Fixed

  • rypipe-python crate README. The crate shipped without a README.md, so its crates.io page reported a missing file. Added the README plus the manifest readme, homepage, documentation, keywords, and categories fields.
  • Stale rypipe-core README references. The dependency example now reads 0.5, and the pinned Arrow version was corrected from =59.2.0 to =59.3.0.

CI

  • rypipe-python trusted publishing. rypipe-python now publishes to crates.io through OIDC trusted publishing, like rypipe-core; this is the first release to use it.

0.5.0 - 2026-10-01

Added

  • Incremental streaming parsers (streaming-parser feature). StreamingRecordParser + RecordStream let an adapter consume arbitrarily chunked input (including non-seekable feeds) and emit adapter-defined records one at a time, with a ParseDiagnostics channel for recoverable events (malformed record, truncated input). This surface was extracted from the xmlstreamer adapter, whose streaming engine previously lived outside rypipe; the engine now owns the buffer, compaction, EOF handshake and diagnostics, while the adapter keeps its format-specific scanner (section policy, delimiter validation, item isolation).
  • Collecting diagnostics sink. CollectingDiagnostics implements ParseDiagnostics, collecting warnings and counters behind a has_events() cheap poll and a take_events() drain, so streaming drivers do not allocate a list per record. ParseDiagnostics gains a default has_events(). The callback signature and the cheap-poll-and-drain shape were taken from the xmlstreamer adapter's diagnostics sink (has_logs()), promoted into core so every streaming adapter gets the same low-overhead path.
  • Stage profiler. rypipe_core::profiling records per-stage timing (split, validate, parse, export, merge) across all workers; disabled by default (a relaxed atomic load per stage). Exposed to Python as rypipe.reset_stage_profile(), rypipe.disable_stage_profile(), and rypipe.stage_profile(), so adapter authors can see where a read spends its time without a system profiler.
  • Chunk-size floor knob. ExecutionPlan gains an opt-in min_chunk_bytes (and with_min_chunk_bytes), with a matching min_chunk_bytes plan kwarg in the Python bindings and adapter readers. It overrides the default 2 MiB split-planning floor so adapters can tune the chunk count for heterogeneous CPUs and small/large files; the default is unchanged. Engines call the new Splitter::find_split_points_with, which defers to an adapter's find_split_points at the default floor. Requested by the xmlstreamer adapter while tuning a laptop CPU with 2 performance cores and 8 efficiency cores; xmlstreamer now forwards the kwarg through every read entry point.

Fixed

  • Duplicate leading split point on tiny inputs. When bytes.len() was smaller than the requested chunk count, every nominal offset collapsed to 0 and the prepended leading split point duplicated it, violating the documented split contract (sorted, unique, first 0, last bytes.len()). Zero candidates are now dropped before the prepend.
  • Unbounded batch count under budgets smaller than one row. A budget below a single input row collapsed rows_per_batch to 1, so the bounded executor emitted one batch per row and consumers that concatenate batches (such as arrow::ConcatenateTables) degraded quadratically (10 MB at a 1 KB budget took about 12 s versus 0.4 s unbounded). The oversize regime now floors rows_per_batch at 64, skips the early flush, and only emits on the row-count condition; the same workload runs in about 0.03 s.

Documentation

  • Plan and API references updated for min_chunk_bytes. The plan architecture, adapter walkthrough, fusion notes, memory and chunking guide, and the Python and Rust API references document the new floor, its default, and the Python kwarg mapping.

CI

  • crates.io publishing for rypipe-python. The release workflow now publishes rypipe-python after rypipe-core, so adapter crates can depend on the bindings as a registry dependency instead of a local path. rypipe-python is still the PyO3 source of the rypipe PyPI wheel, not a separate Python distribution.

0.4.0 - 2026-09-16

Added

  • budget module: BudgetPartition, BudgetLedger, StreamStats. New public module for memory-budget accounting. BudgetPartition derives per-component limits from a MemoryBudget with proof tests that the worst case sums within budget; BudgetLedger tracks capacity-accurate live bytes and their peak; StreamStats reports batches, rows, oversize_batches, and peak_tracked_bytes per streaming run. BoundedExecutor gains run_stream_with_stats and run_bytes_stream_with_stats; sizing and enforcement are unchanged.
  • Strict memory budget contract. MemoryBudget.with_strict(true) makes the budget a hard limit: capacity-accurate per-builder checks via TableBuilder::set_memory_budget error with Error::Memory { used, limit } when an allowance is exceeded. Parallel streaming derives per-worker chunk size and slot allowances from the budget.
  • Memory errors map to Python MemoryError. The engine memory error now surfaces as MemoryError in the bindings instead of a generic exception.
  • Declarative stateful parsing. Splitter gains RecordBoundary, continuation_char, and comment_prefixes for multi-line record formats (line continuations, comment lines), plus find_next_record_boundary and is_continued_newline helpers.
  • _rypipe.errors module. Typed exception module for the Python bindings.
  • Fuzz targets and seed corpora. cargo-fuzz targets with documented seeds and CI smoke campaigns.
  • engine_probe example and benchmark harness update. Probe binary for engine internals, with recorded memory and boundary benchmark results.
  • Expression API string transforms. Added strip, lstrip, rstrip, lower, upper, replace, and length methods to the col() expression API. These produce fusable filter spec dicts that run in the Rust parse loop.
  • Partial pushdown in CastTypes. Non-pushable type casts (e.g. custom Python types) are skipped during plan fusion instead of falling back to a full Python pass over the table.
  • Fused is_type, is_null, contains ops. _fuse_filter_spec now handles is_type, is_null, and contains operators, keeping them in the Rust parse loop instead of falling back to Python.

Security

  • Decompression bomb protection. Added 1 GiB limit on decompressed output via LimitReader. Compressed inputs (gzip, zstd, lz4) that would decompress beyond 1 GiB are rejected mid-stream to prevent OOM kills.
  • Regex size limit. Compiled regex patterns are capped at 1 MiB to reduce ReDoS risk and memory consumption.
  • Supply chain audit. Added cargo-deny advisory/license/ban checks to CI. Pinned all GitHub Actions to full commit SHAs.
  • In/NotIn values list cap. Values lists are capped at 100,000 elements to prevent excessive memory allocation and O(n) per-row scan amplification.
  • Replace filter empty-old rejection. Replace filter with empty old string is rejected to prevent per-row memory amplification (old="" inserts new before every character).
  • Decimal128 scale cap. Scale values are capped at 38 (Arrow maximum) to prevent i8 overflow in the Arrow schema.

Fixed

  • Filter comparisons on unparseable values. Typed filter comparisons (Int64, Float64) now fail when values cannot be parsed, instead of silently comparing as 0/0.0. Division by zero in ArithOp::Div now returns false instead of producing infinity/NaN.
  • Dictionary UTF-8 corruption. Dictionary merge paths use from_utf8_lossy instead of unwrap_or("") to preserve corrupt data with replacement characters instead of silently creating empty-string entries.
  • HashMap unwrap panics in dictionary unification. dict.rs and parallel.rs HashMap lookups now use graceful fallback instead of panicking on missing keys from corrupted input.
  • Dictionary code bounds checks. Dictionary code indexing in columnar.rs now returns None for out-of-bounds codes instead of panicking.
  • Bounds checks on find_literal and in_skip_region. Prevent panics when at > hay.len() or when opener extends past the target position.
  • Empty ranges from degenerate splitter. bounded.rs and parallel.rs now guard against empty chunk ranges, matching the existing guard in parallel_stream.rs.
  • RowObserver panic safety in bounded path. Observer hooks in the bounded parse path are now wrapped in catch_unwind, matching the parallel path behavior.
  • FieldType::FromStr error type. Changed from () to String for actionable error messages that include the bad input.
  • LimitReader error message. Decompression bomb error now includes the actual byte count.
  • threads=0 rejected. resolve_engine now rejects threads=0 with a clear error.
  • parse_memory_string validation. Empty, negative, and non-finite memory strings are now rejected.
  • Plan parameter validation. auto_dict_threshold (must be 0.0-1.0), auto_dict_max_size (must be >= 1), and max_split_chunks (must be >= 1) are now validated.
  • StrColumn::with_capacity cap. Allocation is capped at 1M rows to prevent multi-gigabyte allocations from bad row-size estimates.
  • Overflow-safe chunk splitting. bytes.len() * i / n replaced with bytes.len() / n * i to prevent integer overflow on large files.
  • TOCTOU mitigation. Path is canonicalized before file reopen in bounded stream to prevent symlink swap between mmap planning and reading.
  • RenameFields target-name collisions. Raises ValueError when multiple source fields map to the same target name, which previously caused silent data loss.
  • isin()/not_in() bare string guard. Passing a bare string instead of a list no longer panics; raises a clear error instead.
  • ParquetWriter resource leak. to_parquet wraps the writer in try/finally to ensure the file is closed on write errors.
  • Empty range fallback in parallel stream. parallel_stream.rs now uses actual_bytes instead of the raw range length, preventing empty chunks from degenerate input.
  • Redundant ternary in read_file. Removed dead conditional in the file-reading path.

Changed

  • Poisoned lock recovery. Schema cache and parallel stream locks now recover from poisoned state via unwrap_or_else(|e| e.into_inner()) instead of panicking.
  • Unsafe get_unchecked removed. Dictionary code remapping in columnar.rs and dict.rs uses bounds-checked get() instead of unsafe get_unchecked.
  • warn(unsafe_code) lint. Added to rypipe-core crate root. Removed blanket #![allow(unsafe_code)] from input.rs.
  • Regex size limit raised. From 256 KiB to 1 MiB to reduce false positives on legitimate patterns.
  • Dead code removed. Legacy run_mapped and apply_plan_filter functions deleted from bounded.rs. Dead _arrow_iter removed from fusion.py.
  • Dictionary unification performance. Removed unnecessary clone of column_order during per-chunk dictionary unification in parallel.rs.
  • FFI error propagation. unwrap() at the Python FFI boundary in plan_kwargs replaced with PlanError for actionable error messages.

Documentation

  • Docs overhaul. Reworked tutorial, building-adapters, advanced, and architecture sections; compact top-level nav with dropdowns.
  • Integrations section. New guides for Arrow, DuckDB, Polars, pandas, Parquet, Delta Lake, Iceberg, Spark, direct SQL databases, and Excel.
  • Examples section. Benchmark pages, engine probe walkthrough, report fixture, and three example adapters (INI, LDIF, Properties).
  • Memory contract documented. What the budget covers, what it does not, and how strict mode behaves.
  • Page titles fixed. Front-matter titles across all pages; the homepage no longer shows a code comment as its browser tab title.

CI

  • Adapter and template job. Native example adapters are built and tested as wheels; the cargo-generate template is generated and smoke-tested.
  • Linux ARM64 regression job. Native Rust tests now also run on ARM64.
  • Publishing workflows. Fixed the pinned rust-toolchain invocation to pass the now-required toolchain input.

0.3.1 - 2026-09-11

Fixed

  • Flaky schema cache test. Removed global counter assertions from schema_cache_hits_misses_and_plan_changes that were affected by parallel test interference.

Changed

  • Publishing CI: test blocker. Both PyPI and crates.io publishing workflows now run tests as a required gate before publishing.

0.3.0 - 2026-09-11

Added

  • Expression API with col() for fusable filter predicates. col("amount") > 100 builds a predicate that fuses into the Rust parse loop, just like keyword-form filters. Supports & (and), | (or), ~ (not) composition, and methods: isin, not_in, startswith, endswith, contains, matches, between, is_null, is_not_null, is_type.
  • FilterRowsAny, FilterRowsAll, FilterRowsNot combinators. Build OR, AND, and NOT logic over keyword-form filters. Combinators are fusable and nestable: FilterRowsAny(A, FilterRowsAll(B, FilterRowsNot(C))) expresses A or (B and not C).
  • RowObserver trait and Python observer hooks. Per-row instrumentation fired during parsing. The Rust RowObserver trait has methods on_begin_row, on_put_field, on_row_accepted, on_row_rejected, on_chunk_finished. Python passes observer={"on_row_rejected": fn, ...} in plan kwargs, wrapping callables in a PyObserver.
  • strict_types mode. strict_types=True on Source.__init__ or ExecutionPlan aborts the read when a non-null value fails to parse into its declared type, instead of silently nulling. Genuine nulls remain null.
  • max_split_chunks parameter. Caps the number of bounded-streaming chunks. Defaults to 100,000. Exposed as max_split_chunks on Source.__init__ and ExecutionPlan.
  • MemoryBudget struct and BatchConsumer trait. MemoryBudget::new(bytes) sets a streaming budget. BatchConsumer is a callback trait for streaming reads without batch collection: Pipeline::read_bytes_stream_consumer(bytes, budget, consumer).
  • parse_decimal128 support. FieldType::Decimal128 and CastTypes now parse decimal values.
  • rypipe-test crate. Property-based testing helpers for adapter development: proptest strategies (arb_field_name, arb_field_value, arb_record, arb_records, arb_malformed_utf8, arb_nested_quotes), fixtures (MALFORMED_UTF8, NESTED_QUOTES, EMPTY_VALUES), and test helpers (parse_test_bytes, assert_batches_equal, KeyValueParser, NewlineSplitter).
  • cargo-generate template for adapter scaffolding. cargo generate emiliano-go/rypipe template creates a ready-to-build adapter package.
  • chunks= kwarg on CrystalXMLSource. Passes through to max_split_chunks for parallel reads.
  • memory= kwarg on all sink functions. collect(pipeline, memory="64MiB"), to_pandas(pipeline, memory=...), to_polars(pipeline, memory=...), to_parquet(pipeline, path, memory=...) all support bounded-memory streaming.
  • discover_schema() function. Scans a file once and returns column names after applying field_mapping, drop_fields, etc. Avoids per-file discovery overhead.
  • Pipeline.to_pandas(), to_polars(), to_parquet() methods. Materialize pipeline results directly without importing sink functions.
  • Binary memory units. iter_record_batches and Source.to_pandas/to_polars/to_parquet accept KiB, MiB, GiB, TiB in addition to KB, MB, GB, TB.
  • starts_with, ends_with, contains filter operators. Fusable keyword-form operators for string prefix, suffix, and substring matching.
  • schema() respects projection. When schema=["Name", "Amount"] is passed, Source.schema() returns ["Name", "Amount"] instead of the full column list.
  • Splitter::find_split_points default method. Finds safe split points via nominal offsets, next_record_start, skip-region rejection, dedup, and sort. Adapters rarely need to override.
  • FrozenSchema::from_partial_plan and is_exact. from_partial_plan builds a schema from a partial column list. is_exact reports whether the schema matches exactly.
  • Schema cache functions. insert_schema_cache, clear_schema_cache, schema_cache_stats, dynamic_window_count, dynamic_window_size, layout_signature for schema discovery optimization.
  • Timestamp format variants. FieldType::Timestamp(TimeUnit) supports Second, Millisecond, Microsecond, Nanosecond, plus custom chrono formats: timestamp[ms,format=%Y%m%d %H:%M].
  • Predicate composable trait. Python objects with _to_spec() returning a spec dict can be passed as FilterRows(predicate=obj).

Fixed

  • ArithmeticCompare.cmp_value type. Was documented as f64, corrected to String.
  • Length.value type. Was documented as i64, corrected to String.
  • Compound filter specs fall back to Python with full plan override. FilterRowsAny/All/Not and unsupported ops (is_null, is_type, regex, comparisons) now correctly apply rename, drop, and cast transformations in the Python fallback path.
  • iter_record_batches memory passthrough. Memory strings are parsed to bytes before passing to the Rust extension, which only accepts integer byte counts.
  • schema() projection. Source.schema() now returns self._schema when set, respecting the projection.

Changed

  • to_dataframe() removed. Use to_pandas() instead. The alias existed for backward compatibility with crxml 2.1.
  • Lambda compiler sunset. The in-tree lambda bytecode compiler is removed. Use col() expressions or keyword-form filters instead; for arbitrary logic, use plain callables (Python fallback, not fusable).
  • ColumnarSink trait expanded. 10 new methods: put_row, resolve_raw, resolve_and_put_raw, put_field_resolved, put_field_at, resolve_and_put, needs_value, needs_resolve, wanted_mask, expect_slot.
  • Pipeline expanded. 3 new methods: read_bytes_stream_consumer, read_path_stream_consumer, read_path_stream_par.
  • ExecutionPlan gained fields. strict_types: bool, max_split_chunks: Option<usize>, observer: Option<Arc<dyn RowObserver>>.

0.2.2 - 2026-09-04

Added

  • Contiguous dictionary storage. Dictionary-encoded columns now use contiguous storage for better cache locality and compression.

Fixed

  • Pipeline examples use CrystalXMLSource instead of rypipe.read in documentation.
  • Hello World adapter example tested and working.
  • Prose text cleanup. Replaced : with , across 4 files.
  • Documentation fixes. Removed redundant Pages table, fixed missing blank lines before numbered lists, updated performance numbers in why-python.

0.2.1 - 2026-09-03

Fixed

  • wanted_mask projection short-circuit. Projection short-circuit fired incorrectly when field_map or drop_fields were active.
  • Cross-platform is_x86_feature_detected! guard. Added cfg guard for non-x86 platforms.
  • Dead code on RowBuffer::new. Allowed dead_code to fix clippy error.
  • bench_tier return type. Corrected from (f64, usize) to (f64, f64, usize).
  • _headers file for correct Content-Type on sitemap.xml.
  • Writing Adapters nav points to directory, not missing .md file.

Changed

  • Documentation expanded. Added tagline, "Why rypipe", "What rypipe is not" sections. Fixed 6 audit findings.

0.2.0 - 2026-09-02

Added

  • SIMD scan module with runtime dispatch. scan/ module with AVX2/SSE NEON detection; the engine picks the fastest path at runtime.
  • Tier ladder in core. Performance tiers behind the bench feature for benchmarking individual optimizations.
  • Splitter default method + SkipRegionFinder + chunk floor. find_split_points has a complete default implementation. SkipRegionFinder defines byte ranges that must not be split on. plan_chunk_count enforces a minimum chunk size.
  • dict.rs incremental dictionary unification. Builds dictionary-encoded columns incrementally across chunks.
  • Parallel streaming + BatchConsumer + StreamingBatchIterator. True parallel bounded-memory streaming via read_path_stream_par.
  • put_field_at + expect_slot/record_slot/layout_broken on ColumnarSink. Slot-based fast path for adapters that verify field identity via expect_slot + memcmp.
  • PrimColumn<T> + validity bitmap. Replaces Vec<Option<T>> with packed storage and a bitpack validity array.
  • BlockMasks engine asset. Precomputed bitmasks for fast column membership testing.
  • Frozen schema + parallel discovery. FrozenSchema resolves column order and types once; parallel discovery scans multiple windows concurrently.
  • 64KB streaming via BatchConsumer + StreamingBatchIterator. True streaming at 64KB memory budget with dynamic bytes_used and split_off.
  • Cow-based Value::Str. Value::Str(Cow<'a, str>) borrows from input bytes, closing filtered-path use-after-free.
  • Advanced docs section. 2000+ lines covering architecture, fusion, streaming, profiling, parallelism, memory, I/O tuning, schema, stage protocol, source patterns, adapter design, case studies.

Fixed

  • plan_chunk_count hardcoded chunks. Was hardcoded to 1; now uses max_chunks.
  • in_skip_region overhead. Uses memchr::memmem::find directly instead of Finder::new.
  • Clippy warnings. Resolved all clippy -D warnings for CI.
  • starts_with out-of-bounds panic. Fixed bounds check.
  • Parallel stream channel types. Corrected channel type parameters.
  • Reorder buffer overflow fallback. Falls back to unordered delivery when the buffer overflows.
  • reset_child_ordinal syncs adapter/engine ordinal counters.

Changed

  • engine.rs (2392 lines) split into engine/ module. Modular structure for maintainability.
  • Arrow bumped to 59.2.0, pyo3 to 0.29.
  • Documentation rewritten. Architecture (2000 lines) and writing-adapters (7 pages) fully rewritten.

0.1.1 - 2026-08-23

Fixed

  • crates.io metadata and trusted publishing. Added LICENSE, README, and description to Cargo.toml for crates.io publishing.

0.1.0 - 2026-08-23

Added

  • Format-agnostic columnar engine. Core rypipe-core crate with Splitter, RecordParser, ColumnarSink traits, Pipeline high-level API, ExecutionPlan builder, and Arrow export.
  • Crystal Reports XML adapter (crxml). Reference adapter demonstrating the full trait implementation.
  • PyO3 bindings (rypipe-python). execution_plan_from_kwargs, record_batches_to_pyarrow_table, record_batch_to_pyarrow, exception types (ParseError, XmlError, PlanError, MergeError).
  • Public rypipe package. Source, Pipeline, stages (RenameFields, DropFields, CastTypes, FilterRows), sinks (collect, to_arrow, to_pandas, to_polars, to_parquet, to_csv), read(), read_par(), read_stream(), read_batches(), iter_record_batches(), register_adapter(), resolve_engine().
  • Adapter base class. One-method source creation: subclass Adapter, implement read(), get caching and pipelines for free.
  • abi3 wheel builds. Stable ABI wheels for Python 3.10+.
  • aarch64 manylinux cross-compilation with zig.
  • Zensical docs site. Tutorial, building-adapters guide, architecture, advanced topics, reference pages.
  • Benchmark harness. Throughput benchmarks for the columnar engine.