Add builder to help create Schemas for shredding (`ShreddedSchemaBuilder`) by XiangpengHao · Pull Request #8940 · apache/arrow-rs

XiangpengHao · 2025-12-01T16:01:45Z

Closes [Variant] easier way to construct a shredded schema #8922

Rationale for this change

Basically a helper to simplify this:

let shredding_type = ShredTypeBuilder::default()
    .with_path("a", &DataType::Int64)
    .with_path("b.c", &DataType::Utf8)
    .with_path("b.d", &DataType::Float64)
    .build();

assert_eq!(
    shredding_type,
    DataType::Struct(Fields::from(vec![
        Field::new("a", DataType::Int64, true),
        Field::new(
            "b",
            DataType::Struct(Fields::from(vec![
                Field::new("c", DataType::Utf8, true),
                Field::new("d", DataType::Float64, true),
            ])),
            true
        ),
    ]))
);

What changes are included in this PR?

Added ShredTypeBuilder
Updated existing tests cases to use this new primitive

Are these changes tested?

Yes

Are there any user-facing changes?

Add a new public interface

klion26

Thanks for the contribution, LGTM, left some nit comments to be considered.

klion26 · 2025-12-02T07:50:58Z

+                if child_fields.is_empty() {
+                    None
+                } else {
+                    Some(DataType::Struct(Fields::from(child_fields)))


Seems we can construct DataType::Struct and primitive types (use "" as the path), and no List types, not sure if we need to add the information struct information in the builder name or anywhere.

I am not sure what you mean by this

ShredTypeBuilder is primarily used for building Struct data types. Since Variants can be shredded as either Structs or Arrays, the previous comment raises the question of whether we should inform users that this cannot be used to build Array schemas.

Added a note: 6856ada

klion26 · 2025-12-02T11:07:23Z

+    }
+
+    #[test]
+    fn test_variant_schema_builder_conflicting_path() {


Nice tests!

alamb · 2025-12-03T15:10:52Z

FYI @mhilton -- this PR may be interesting to you

alamb

Thanks @klion26 and @XiangpengHao -- this is great

The only thing I think we should consider is supporting FieldRefs directly rather than assuming DataType / nullable fields.

I left some other comments that might be useful but are not necessary in my opinion

alamb · 2025-12-03T17:40:35Z

+                if child_fields.is_empty() {
+                    None
+                } else {
+                    Some(DataType::Struct(Fields::from(child_fields)))


I am not sure what you mean by this

alamb · 2025-12-03T17:41:16Z

-            Field::new("age", DataType::Int64, true),
-        ]));
+        let schema2 = ShredTypeBuilder::default()
+            .with_path("id", &DataType::Int32)


this certainly is much nicer now

alamb · 2025-12-03T17:43:01Z

FYI @scovich in case you have seen similar APIs or have ideas

mhilton

I think it would be nice if this builder had a similar structure to VariantBuilder with new_object and new_list. A list can only have one type so it wouldn't work exactly the same though. However such an API and the proposed API aren't mutually exclusive.

mhilton · 2025-12-04T13:34:56Z

+
+/// Internal tree node structure for building variant schemas.
+#[derive(Clone)]
+enum VariantSchemaNode {


It'd be nice if this type also supported an Array type too. Although the path syntax would need to be extended a little. Possibly you could use .0 to indicate an array?

Arrow List is not supported by shred_variant yet, maybe we can wait until array shredding is implemented

I added a note here: 6856ada

scovich · 2025-12-04T21:37:51Z

Traveling this week but would love to take a look when I get a chance, hopefully next week

alamb · 2025-12-10T17:18:18Z

I just pushed a fix for the test failures https://github.com/apache/arrow-rs/actions/runs/19977045867/job/57295616147

failures:

---- shred_variant::tests::test_variant_schema_builder_empty_path stdout ----

thread 'shred_variant::tests::test_variant_schema_builder_empty_path' (3091) panicked at parquet-variant-compute/src/shred_variant.rs:1675:9:
assertion `left == right` failed
  left: Struct([Field { name: "", data_type: Int64, nullable: true }])
 right: Int64
stack backtrace:
   0: __rustc::rust_begin_unwind

The issue was that VariantPath::from("") was resulting in VariantPath::new(vec![""]) rather than VariantPath::new(vec![])).

I pushed a fix in 1081990

alamb · 2025-12-10T17:23:12Z

In terms of List shredding, it seems like perhaps after this PR from @liamzwbao

[Variant] Support array shredding into List/LargeList/ListView/LargeListView #8831

it would be a good follow on project (could someone file a ticket)?

# Which issue does this PR close? - Related to #8940 # Rationale for this change When reviewing #8940 it wasn't clear you could create a `VariantPath` from a string Thus let's add an example to the documentation # What changes are included in this PR? Add an example to the documentation # Are these changes tested? Yes, by CI # Are there any user-facing changes? Docs only, no functional change

scovich · 2025-12-10T23:02:20Z

The issue was that VariantPath::from("") was resulting in VariantPath::new(vec![""]) rather than VariantPath::new(vec![])).

I pushed a fix in 1081990

Good catch... I think that was probably my bug 🤦

alamb · 2025-12-11T19:33:31Z

Thanks @XiangpengHao @scovich @mhilton and @klion26 -- we are moving this right along again

XiangpengHao added 3 commits December 1, 2025 09:57

update

92b18e8

nullable

9bfc5ac

fmt

35e7d59

github-actions Bot added the parquet-variant parquet-variant* crates label Dec 1, 2025

klion26 approved these changes Dec 2, 2025

View reviewed changes

alamb approved these changes Dec 3, 2025

View reviewed changes

alamb changed the title ~~Add variant ShredTypeBuilder~~ Add builder to help create Schemas for shredding (ShredTypeBuilder) Dec 3, 2025

alamb mentioned this pull request Dec 3, 2025

Add example of parsing field names as VariantPath #8945

Merged

mhilton reviewed Dec 4, 2025

View reviewed changes

address comments

f11ec5b

XiangpengHao changed the title ~~Add builder to help create Schemas for shredding (ShredTypeBuilder)~~ Add builder to help create Schemas for shredding (ShreddedSchemaBuilder) Dec 5, 2025

alamb added 2 commits December 10, 2025 12:06

Merge remote-tracking branch 'apache/main' into variant-schema

71480ec

Fix clippy and tests

1081990

Add doc note about struct shredding

6856ada

fix docs

823430b

alamb merged commit cf8582b into apache:main Dec 11, 2025
17 checks passed

alamb mentioned this pull request Jan 6, 2026

[Variant] easier way to construct a shredded schema #8922

Closed

Conversation

XiangpengHao commented Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

klion26 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb commented Dec 3, 2025

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb commented Dec 3, 2025

Uh oh!

mhilton left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

scovich commented Dec 4, 2025

Uh oh!

alamb commented Dec 10, 2025

Uh oh!

alamb commented Dec 10, 2025

Uh oh!

scovich commented Dec 10, 2025

Uh oh!

Uh oh!

alamb commented Dec 11, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

XiangpengHao commented Dec 1, 2025 •

edited

Loading