ArrowJsonBatch::from_batch is incomplete

[phil-opp]

Describe the bug

The arrow_integration_test::ArrowJsonBatch::from_batch only supports DataType::Int8. All other data is translated to a column with no data.

This limitation is not mentioned anywhere in the docs.

To Reproduce

See https://docs.rs/arrow-integration-test/latest/src/arrow_integration_test/lib.rs.html#1054

Expected behavior

Either an error or support for all data types.

Additional context

We want to use this function to store a human-readable representation of some arrow arrays. We need to support fields containing binary data, so the arrow_json crate is not suitable for our use case.

[tustvold]

This crate exists to serve the needs of the arrow integration tests, it isn't meant for usage beyond this.

With regards to binary data, arrow_json does support binary data by hex encoding it.

[phil-opp]

Thanks for the quick reply!

This crate exists to serve the needs of the arrow integration tests, it isn't meant for usage beyond this.

Perhaps some kind of disclaimer that specifies this would be nice? Right now the crate description specifies:

Support for the Apache Arrow JSON test data format

The linked document even describes all the data types that are supported in that format. I don't see the limitation to int8 anywhere and neither do I see a "please don't use this crate or format" note.

Given that our use case is integration testing with arrow input as well, the crate seemed like the logical choice.

With regards to binary data, arrow_json does support binary data by hex encoding it.

Could you give some more details on this? The arrow-json documentation specifies quite explicitly that binary data is not supported:

Binary Data

As per RFC7159 JSON cannot encode arbitrary binary data. A common approach to workaround this is to use a binary-to-text encoding scheme, such as base64, to encode the input data and then decode it on output.

After checking the code is seems to be able to convert binary Arrow arrays to JSON. But not the other way around, i.e. JSON data to binary Arrow arrays:

arrow-rs/arrow-json/src/reader/mod.rs

Lines 746 to 748 in a7572eb

	DataType::Binary | DataType::LargeBinary | DataType::FixedSizeBinary(_) => {
	Err(ArrowError::JsonError(format!("{data_type} is not supported by JSON")))
	}

Or am I missing anything?

[tustvold]

The linked doc https://docs.rs/arrow-json/latest/arrow_json/#binary-data demonstrates how to roundtrip binary data through JSON.

[phil-opp]

Not really, it shows you that you need to add a second (de)serialization step because binary data is not supported.