DriverVectorCube to GeoParquet: unsupported driver: 'Parquet'

[bossie]

From this forum post: https://discuss.eodc.eu/t/read-and-use-geoparquet-vector-data/771

Process graph:

{
  "process_graph": {
    "load1": {
      "process_id": "load_url",
      "arguments": {
        "format": "PARQUET",
        "url": "https://github.com/joheisig/public_data/raw/main/test_gpd.parquet"
      }
    },
    "save2": {
      "process_id": "save_result",
      "arguments": {
        "data": {
          "from_node": "load1"
        },
        "format": "PARQUET"
      },
      "result": true
    }
  }
}

Error:

Traceback (most recent call last):
  File "batch_job.py", line 489, in <module>
    main(sys.argv)
  File "batch_job.py", line 181, in main
    run_driver()
  File "batch_job.py", line 152, in run_driver
    run_job(
  File "/opt/venv/lib64/python3.8/site-packages/openeogeotrellis/utils.py", line 56, in memory_logging_wrapper
    return function(*args, **kwargs)
  File "batch_job.py", line 292, in run_job
    the_assets_metadata = result.write_assets(str(output_file))
  File "/opt/venv/lib64/python3.8/site-packages/openeo_driver/save_result.py", line 174, in write_assets
    return self.cube.write_assets(directory=directory, format=self.format, options=self.options)
  File "/opt/venv/lib64/python3.8/site-packages/openeo_driver/datacube.py", line 579, in write_assets
    gdf.to_file(path, driver=format_info.fiona_driver, crs=self.get_crs())
  File "/opt/venv/lib64/python3.8/site-packages/geopandas/geodataframe.py", line 1263, in to_file
    _to_file(self, filename, driver, schema, index, **kwargs)
  File "/opt/venv/lib64/python3.8/site-packages/geopandas/io/file.py", line 572, in _to_file
    _to_file_fiona(df, filename, driver, schema, crs, mode, **kwargs)
  File "/opt/venv/lib64/python3.8/site-packages/geopandas/io/file.py", line 598, in _to_file_fiona
    with fiona.open(
  File "/opt/venv/lib64/python3.8/site-packages/fiona/env.py", line 457, in wrapper
    return f(*args, **kwds)
  File "/opt/venv/lib64/python3.8/site-packages/fiona/__init__.py", line 303, in open
    colxn = Collection(
  File "/opt/venv/lib64/python3.8/site-packages/fiona/collection.py", line 211, in __init__
    raise DriverError("unsupported driver: %r" % driver)
fiona.errors.DriverError: unsupported driver: 'Parquet'

[bossie]

From fiona 1.9.4 CHANGELOG (we use the latest, currently 1.9.6):

The Parquet driver has been added to the list of supported drivers and will be available if your system’s GDAL library links libarrow. Note that fiona wheels on PyPI do not include libarrow as it is rather large.

Because of the way fiona is currently installed, "your system’s GDAL library" means: the GDAL that fiona itself installs, as described in Easy installation; it is not the system-wide gdal that is part of the Docker image.

fiona's GDAL does not have support for Parquet.

We could build our own gdal RPM with support for Parquet, then install fiona from source and point it to this gdal as described in Advanced installation.

In practice, this seems to mean that these dependencies should be available when building the gdal RPM: arrow-devel and parquet-devel as described here.

[bossie]

An easy way to check which drivers fiona supports is:

import fiona
fiona.supported_drivers.keys()

[VincentVerelst]

Hi @bossie,

In my naivity it seems that adding something like this:

if format_info.format == 'parquet':
    gdf.to_parquet(...)
else:
    gdf.to_file(...)

at https://github.com/Open-EO/openeo-python-driver/blob/master/openeo_driver/datacube.py#L594 could be a quick fix. This uses the PyArrow driver. The same logic is used in AggregatePolygonResult.write_assets(): https://github.com/Open-EO/openeo-python-driver/blob/master/openeo_driver/save_result.py#L611

Is there a reason you are avoiding it here?

[bossie]

Nope, just an oversight. Seems to work 👍