[C++][CSV] Add support for Extention type in csv reader

[asfimport]

Reporter: Artem Alekseev / @fexolm

Related issues:

• [C++][Python] Support ExtensionType arrays in more kernels (is related to)

PRs and other links:

• GitHub Pull Request #5880

_{Note: This issue was originally created as ARROW-7085. Please see the migration documentation for further details.}

[asfimport]

Wes McKinney / @wesm:
@fexolm could you clarify what you need – a custom ColumnBuilder? @pitrou should be able to advise you about this

[asfimport]

Artem Alekseev / @fexolm:
@wesm, @pitrou  For our purposes we need to define a custom type and use arrow CSV reader to parse it. Unfortunately, there is no way to use custom types with CSV reader right now, because of type dispatch when choosing an appropriate column builder. 

I'm not sure how architecturally it could be solved, maybe custom ColumnBuilder is a case.

[asfimport]

Antoine Pitrou / @pitrou:
cc @fexolm Do you need to define a custom converter?

At worse, you could parse the column as string or binary and do the casting afterwards (if that's not too costly for you).

[asfimport]

Wes McKinney / @wesm:
It may make sense to start with a proposal about what kind of public API would be desirable and then work backwards towards the implementation strategy.

[asfimport]

Antoine Pitrou / @pitrou:
Right. Though if casting is acceptable the problem is pretty much solved ;-)

[asfimport]

Artem Alekseev / @fexolm:
@pitrou  We need to encode strings with custom encoder and avoid the overhead of parsing column as a string and converting it to integers.

I think there are 2 options of desirable API:

• add an option to define custom Converter for ExtensionType (for now it's impossible to use extension types in CSV reader)

• add an option to define custom ColumnBuilder for a specific column

  I think such API can be useful, but for my problem, I found that arrow dictionary encoding can do the job. But I'm not sure if there is a way to e.g. check if a specified string is in the dictionary, or how to get string index. Does Arrow provide such API?

   

[asfimport]

Antoine Pitrou / @pitrou:
No, Arrow does not provide an API to find out the index of a string in the dictionary. The dictionary is a simple linear array, so you can still do a linear scan for the value you're interested in.

[asfimport]

Antoine Pitrou / @pitrou:

We need to encode strings with custom encoder and avoid the overhead of parsing column as a string and converting it to integers.

You mean you just need to parse them as regular integers? I'm not sure I understand: are your columns using a custom format?

[asfimport]

Artem Alekseev / @fexolm:
@pitrou  we use our own dictionary encoding

[asfimport]

Antoine Pitrou / @pitrou:
If extension types can define their own cast functions (see ARROW-5890), this could be implemented by casting from string to the given extension type.

[github-actions]

This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.