Splitting large datasets

For most tasks, snail split and snail process commands should just work. Or, using the Python functions snail.intersection.split_linestrings(), split_polygons() and split_geometries() you can pass in a GeoDataFrame of geometries and get a split one back.

This page goes into more of the implementation details, which might help if you want to split a source that is not already loaded as a GeoDataFrame - for example, a GeoParquet file bigger than available memory - or if you are working directly against the compiled extension.

How a split runs

The splitting logic is all implemented in a C++ extension, which is compiled and made available to Python as :module:`snail.core.intersections`.

Geometries are passed into the C++ extension as GeoArrow, using the Arrow C stream interface: the extension pulls one batch of geometries at a time, splits it, and hands the pieces back the same way, as a stream of record batches. Nothing is split until that stream is read, and only one batch of the source is ever held at once - so it should be possible for a file larger than memory to be split by a consumer that takes the results as they come, without loading the whole thing first.

snail.core.intersections.split_linestrings(), split_polygons() and split_geometries() accept the geometries from any object implementing the Arrow PyCapsule stream interface (__arrow_c_stream__) or the single-array interface (__arrow_c_array__, read as a stream of one batch) - a pyarrow.ChunkedArray, Table or RecordBatchReader, a GeoParquet or pyarrow.dataset.Dataset reader, or GeoSeries.to_arrow().

Geometry encoding

GeoArrow specifies two possible encodings, native and well-known binary. Either can be read:

Native - geoarrow.linestring, geoarrow.polygon - keeps the coordinates in Arrow buffers. They may be interleaved (as geopandas exports them) or separated into x and y arrays (as GeoParquet stores them), in two or more dimensions; splitting works on x and y dimensions, so a z or m co-ordinates are stepped over.

Well-known binary - geoarrow.wkb - holds each geometry as a serialised blob, which has to be decoded on the way in. This is what GeoDataFrame.to_parquet() writes unless told otherwise, so it is what a GeoParquet file usually contains.

Geometries of more than one type

snail.core.intersections.split_geometries() can handle a stream of mixed geometry types:

  • LineStrings and Polygons are split

  • Points pass through

  • multi-part geometries are split part by part

  • GeometryCollections are split member by member

This returns geoarrow.wkb, because an Arrow stream has one schema for all of its batches and WKB is the only encoding that can carry every type in one. That is also why a mixed layer cannot be written as native GeoArrow - geopandas raises ValueError: Geometry type combination is not supported.

There is some overhead to encoding and decoding wkb. Use the typed functions where a column really does hold one type, and split_geometries where it does not.

Splitting a file directly

This reads a GeoParquet file of linestrings in batches and splits each batch as it arrives, never holding more than one batch of geometries and one batch of pieces at a time - whatever the file’s size:

import pyarrow
import pyarrow.parquet
from snail.core.intersections import split_linestrings
from snail.intersection import GridDefinition

grid = GridDefinition.from_raster("hazard.tif")

parquet_file = pyarrow.parquet.ParquetFile("edges.geoparquet")
geometry_batches = pyarrow.RecordBatchReader.from_batches(
    parquet_file.schema_arrow, parquet_file.iter_batches(batch_size=10_000)
)

stream = split_linestrings(
    geometry_batches, nrows=grid.height, ncols=grid.width, transform=grid.transform
)

reader = pyarrow.RecordBatchReader._import_from_c_capsule(stream.__arrow_c_stream__())
for batch in reader:
    # batch has a "geometry" column of the pieces (GeoArrow-encoded)
    # and a "parent" column: the index, in the source, of the
    # geometry each piece was split from
    ...

Swap split_linestrings for split_geometries() to split a file whose geometries are not all one type; the "geometry" column of each batch is then geoarrow.wkb rather than native GeoArrow.

To get the pieces as shapely geometries instead, convert a batch with geopandas.GeoDataFrame.from_arrow().

Or, to read the whole stream into memory, use snail.intersection.read_split_stream().