Column/field description

[simonaubertbd]

Describe the enhancement requested

Hello,

I may have misunderstood or misread the documentation but I didn't find it. I would love to have a field/column description just like we have in databases :

Best regards,

Simon

Component(s)

No response

[gaborcsardi]

For every column, you can store arbitrary key-value metadata (string -> string), see at

parquet-format/src/main/thrift/parquet.thrift

Line 809 in 5a5c894

8: optional list<KeyValue> key_value_metadata

[simonaubertbd]

Hello @gaborcsardi Yes, thanks for the tip but it can't be exploited by a software such as dataviz, etl or dataviz.

[liquidcarbon]

Hi! I'm confused by different types of metadata.

1. there's key-value metadata of type BLOB that you can write via schema metadata (in python pyarrow.table(data, metadata=...)) and read similarly, or through https://duckdb.org/docs/data/parquet/metadata.html#parquet-key-value-metadata
2. there's separate place, with these fields, that you can read with https://duckdb.org/docs/data/parquet/metadata.html#parquet-metadata

['file_name', 'row_group_id', 'row_group_num_rows',
       'row_group_num_columns', 'row_group_bytes', 'column_id', 'file_offset',
       'num_values', 'path_in_schema', 'type', 'stats_min', 'stats_max',
       'stats_null_count', 'stats_distinct_count', 'stats_min_value',
       'stats_max_value', 'compression', 'encodings', 'index_page_offset',
       'dictionary_page_offset', 'data_page_offset', 'total_compressed_size',
       'total_uncompressed_size', 'key_value_metadata']

column-level information and data types are recorded here. But how to you write something into THAT key_level_metadata ?

[simonaubertbd]

Since Parquet is now meant to be a standard file format, use for intermediate storage between tools, it really seems necessary to have this field/column description added. It exists in a lot of format such as Tableau Hyper, Alteryx yxdb, etc

As for now, it means loosing informations when storing in Parquet.

Best regards,

Simon

[gaborcsardi]

Hello @gaborcsardi Yes, thanks for the tip but it can't be exploited by a software such as dataviz, etl or dataviz.

Why can't dataviz or etl software add the description as metadata? E.g. as a key called description?

[simonaubertbd]

@gaborcsardi Because it means asking the user of ETL a key name and the user of the dataviz the key name. This means another place (a file, a mail, an issue...) where the users have to share the information "hey guys, I put the field description in the key label". But since in most ETL or dataviz tool, you don't have the ability to retrieve metadata by key, users of each tools have to ask the tool editor a way to retrieve a value by key description. The whole process isn't user friendly.

Since it's a very common metadata and a lot of file format already have it, I think this is way better to have it in the file format specification. As of today, this is probably my main pain point with Parquet.

Best regards,

Simon

[emkornfield]

@simonaubertbd I don't think this exists in parquet. I'm not against adding it but with the rise in popularity of table formats (which do include description) the utility might not be super high. Is this something you would like to constribute?

[simonaubertbd]

Hello @emkornfield and thanks for the answer.
1/I may disagree about the utility : Parquet has become in the last month a standard for data transfer just like csv (but way better), it's far to be just used on databases and it could even be used to replace proprietary file formats such as Qlik qvd, Alteryx yxdb (so less vendor locking and better interoperability). The only thing missing for that is the field description.

2/I can contribute by testing and having some discussion on specs but I don't even know where to begin. I'm afraid to not have the sufficient skills to develop such a feature.

Best regards,

Simon

[simonaubertbd]

Hello guys,

Any news about that feature?

[emkornfield]

Any news about that feature?

No, it would generally need a volunteer to shepherd this through and nobody has had a willingness to so yet.

[abellgithub]

Why can't dataviz or etl software add the description as metadata? E.g. as a key called description?

Because it means asking the user of ETL a key name and the user of the dataviz the key name.

You could create a data dictionary of predefined keys that readers/writers can create/access if they choose. The file format can remain as it is.

[emkornfield]

You could create a data dictionary of predefined keys that readers/writers can create/access if they choose. The file format can remain as it is.

Yes, one downside is this gets a little awkward because key value metadata for columns today is at a row-group level. which means for file level metadata one would need to define a serialization format for mapping column to description.

[abellgithub]

You could create a data dictionary of predefined keys that readers/writers can create/access if they choose. The file format can remain as it is.

Yes, one downside is this gets a little awkward because key value metadata for columns today is at a row-group level. which means for file level metadata one would need to define a serialization format for mapping column to description.

Could you add optional Key/Value Metadata to SchemaElement? This is more general than a description field. It might also to eliminate the need for some extension type handling.

[emkornfield]

Could you add optional Key/Value Metadata to SchemaElement? This is more general than a description field. It might also to eliminate the need for some extension type handling.

That seems like it could be a reasonable alternative. I would need to refresh my self on the extension type proposal and think about it more deeply.