Allow '-' in parquet column names

[EmileSonneveld]

This graph saves to parquet, but changes Energy-V1-geothermal to Energy_V1_geothermal

process graph

{
"process_graph": {
  "loadstac9": {
    "arguments": {
      "bands": ["geothermal"],
      "url": "https://catalogue.weed.apex.esa.int/collections/Energy-V1"
    },
    "process_id": "load_stac"
  },
  "dropdimension9": {
    "arguments": {"data": {"from_node": "loadstac9"}, "name": "t"},
    "process_id": "drop_dimension"
  },
  "renamelabels14": {
    "arguments": {
      "data": {"from_node": "dropdimension9"},
      "dimension": "bands",
      "source": ["geothermal"],
      "target": ["Energy-V1-geothermal"]
    },
    "process_id": "rename_labels"
  },
  "aggregatespatial1": {
    "arguments": {
      "data": {"from_node": "renamelabels14"},
      "geometries": {
        "features": [
          {
            "geometry": {"coordinates": [10.089101, 59.612353], "type": "Point"},
            "id": "0",
            "properties": {"MGRSid10": "32VNM61450874"},
            "type": "Feature"
          },
          {
            "geometry": {"coordinates": [10.096994, 59.55544], "type": "Point"},
            "id": "1",
            "properties": {"MGRSid10": "32VNM62000241"},
            "type": "Feature"
          }
        ],
        "type": "FeatureCollection"
      },
      "reducer": {
        "process_graph": {
          "first1": {
            "arguments": {"data": {"from_parameter": "data"}},
            "process_id": "first",
            "result": true
          }
        }
      }
    },
    "process_id": "aggregate_spatial"
  },
  "saveresult1": {
    "arguments": {
      "data": {"from_node": "aggregatespatial1"},
      "format": "Parquet",
      "options": {"feature_id_property": "MGRSid10"}
    },
    "process_id": "save_result",
    "result": true
  }
}
}

parquet_path = glob.glob(os.path.join(output_dir, "**/*.parquet"), recursive=True)
table = pq.read_table(parquet_path)
print(table.column_names)  # ['geometry', 'MGRSid10', 'feature_index', 'Energy_V1_geothermal']

[EmileSonneveld]

@bossie I don't find where it is required to sanitize parquet column names. Ok if I just drop it?
Spec: https://geoparquet.org/releases/v1.0.0/#column-metadata
Related ticket: Open-EO/openeo-geopyspark-driver#623

[EmileSonneveld]

When the time dimension is still there, column names are not restricted:

openeo-python-driver/openeo_driver/save_result.py

Line 634 in 8612f3e

gpd.GeoDataFrame(stats.join(gdf, on='feature_index')).to_parquet(filename)

[bossie]

@bossie I don't find where it is required to sanitize parquet column names. Ok if I just drop it?

I wish I included some kind of reference to this naming restriction at the time. Can't find one now either so I would just try it out. AFAICT there are some unit tests that cover this path.

[EmileSonneveld]

There was a unit test that failed when dropping all sanitation. So I jus kept this in the MR as a light sanitation:
‘re.sub(r"[()/\]", "_", name)‘

[bossie]

As far as I remember, it was the parentheses in the column name I had trouble with at the time so I sanitized e.g. avg(band_0) to avg_band_0_.

But apparently without the sanitation, it manages to write Parquet files just fine. Hard to tell how other tools cope with it.

[EmileSonneveld]

So ok to merge? #470

[EmileSonneveld]

In test_spatial_geoparquet, : is slugified, in test_spatiotemporal_vector_cube_to_geoparquet not.
@bossie would it be worth to put behind a parameter? Or just not slugify :?

[EmileSonneveld]

Made a forum post to gather feedback for the change: https://forum.dataspace.copernicus.eu/t/geoparquet-column-names-change/4873