Unify API for writing column chunks / row groups in parallel

[alamb]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

We now have two different APIs for writing row groups in parallel, depending on encryption, and I would like to simplify the code to use just one.

The current example for writing row groups in parallel uses get_column_writers and does not support encryption

@rok and @adamreeve added a new API based on ArrowRowGroupWriterFactory for encoding parquet columns and row groups in parallel, with encryption in

• Enable parallel writing across row groups when writing encrypted parquet #8162,

This API is also somewhat strange in that it makes users create an ArrowWriter only to immediately destructure it into a SerializedWriter / the underlying writer.

The reason we need to expose ArrowRowGroupWriterFactory is that ArrowRowGroupWriterFactory::create_column_writers also has the appropriate encryption properties whereas get_column_writers does not

Describe the solution you'd like
I would like a single easy to use API for writing in parallel that:

1. Is the same for encryption vs not encryption
2. Has clear examples

Describe alternatives you've considered
I suggest:

1. Make the constructors for ArrowRowGroupWriterFactory public
2. Update the example to use ArrowRowGroupWriterFactory / ArrowRowGroupWriterFactory::create_column_writers function
3. Deprecating the existing get_column_writers function directing people to ArrowRowGroupWriterFactory
4. Deprecate ArrowWriter::into_serialized_writer, directing people to ArrowRowGroupWriterFactory

Additional context

[alamb]

both @rok and @adamreeve may have thoughts to share here

[adamreeve]

This approach sounds good to me, I don't have anything to add 🙂

[alamb]

Here is an example of using the ArrowRowGroupWriterFactory api:

• feat: support multi-threaded writing of Parquet files with modular encryption datafusion#16738