oop¶

Module sketching out the OO backbone. This is intended to reduce the complexity of the user facing side of the workflow by wrapping the underlying functions into methods associated to objects that have a defined structure and therefore are more failure robust.

class scalenav.oop.Layer¶

__init__(name='', src='')¶

Validation of the input source goes here. What is the schema ?

Parameters:

name (str)
src (str)

Return type:

None

change_res(levels, transforms=None, **kwargs)¶: Change resolution by a fixed ‘delta’ value. Can be a positive of negative integer. The resulting resolution will be equal to the current one plus the delta value. The restriciton is that the final resolution has to be within the accepted limits.

head(n=5)¶

Wrapper around the head function.

Parameters:: n (int)

join(table2, predicates=None, how='outer')¶

Perform a join between two tables.

Parameters:

right – Right table to join
predicates (list | None) – Condition(s) to join on. See examples for details.
how – Join method, e.g. “inner” or “left”.
lname – A format string to use to rename overlapping columns in the left table (e.g. “left_{name}”).
rname – A format string to use to rename overlapping columns in the right table (e.g. “right_{name}”).

Examples

>>> import ibis
>>> from ibis import _
>>> ibis.options.interactive = True
>>> movies = ibis.examples.ml_latest_small_movies.fetch()
>>> movies.head()
┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ movieId ┃ title                            ┃ genres                          ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ int64   │ string                           │ string                          │
├─────────┼──────────────────────────────────┼─────────────────────────────────┤
│       1 │ Toy Story (1995)                 │ Adventure|Animation|Children|C… │
│       2 │ Jumanji (1995)                   │ Adventure|Children|Fantasy      │
│       3 │ Grumpier Old Men (1995)          │ Comedy|Romance                  │
│       4 │ Waiting to Exhale (1995)         │ Comedy|Drama|Romance            │
│       5 │ Father of the Bride Part II (19… │ Comedy                          │
└─────────┴──────────────────────────────────┴─────────────────────────────────┘
>>> ratings = ibis.examples.ml_latest_small_ratings.fetch().drop("timestamp")
>>> ratings.head()
┏━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┓
┃ userId ┃ movieId ┃ rating  ┃
┡━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━┩
│ int64  │ int64   │ float64 │
├────────┼─────────┼─────────┤
│      1 │       1 │     4.0 │
│      1 │       3 │     4.0 │
│      1 │       6 │     4.0 │
│      1 │      47 │     5.0 │
│      1 │      50 │     5.0 │
└────────┴─────────┴─────────┘

Equality left join on the shared movieId column. Note the _right suffix added to all overlapping columns from the right table (in this case only the “movieId” column).

>>> ratings.join(movies, "movieId", how="left").head(5)
┏━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━┓
┃ userId ┃ movieId ┃ rating  ┃ movieId_right ┃ title                       ┃ … ┃
┡━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━┩
│ int64  │ int64   │ float64 │ int64         │ string                      │ … │
├────────┼─────────┼─────────┼───────────────┼─────────────────────────────┼───┤
│      1 │       1 │     4.0 │             1 │ Toy Story (1995)            │ … │
│      1 │       3 │     4.0 │             3 │ Grumpier Old Men (1995)     │ … │
│      1 │       6 │     4.0 │             6 │ Heat (1995)                 │ … │
│      1 │      47 │     5.0 │            47 │ Seven (a.k.a. Se7en) (1995) │ … │
│      1 │      50 │     5.0 │            50 │ Usual Suspects, The (1995)  │ … │
└────────┴─────────┴─────────┴───────────────┴─────────────────────────────┴───┘

Explicit equality join using the default how value of “inner”. Note how there is no _right suffix added to the movieId column since this is an inner join and the movieId column is part of the join condition.

>>> ratings.join(movies, ratings.movieId == movies.movieId).head(5)
┏━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ userId ┃ movieId ┃ rating  ┃ title                  ┃ genres                 ┃
┡━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━┩
│ int64  │ int64   │ float64 │ string                 │ string                 │
├────────┼─────────┼─────────┼────────────────────────┼────────────────────────┤
│      1 │       1 │     4.0 │ Toy Story (1995)       │ Adventure|Animation|C… │
│      1 │       3 │     4.0 │ Grumpier Old Men (199… │ Comedy|Romance         │
│      1 │       6 │     4.0 │ Heat (1995)            │ Action|Crime|Thriller  │
│      1 │      47 │     5.0 │ Seven (a.k.a. Se7en) … │ Mystery|Thriller       │
│      1 │      50 │     5.0 │ Usual Suspects, The (… │ Crime|Mystery|Thriller │
└────────┴─────────┴─────────┴────────────────────────┴────────────────────────┘

>>> tags = ibis.examples.ml_latest_small_tags.fetch()
>>> tags.head()
┏━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ userId ┃ movieId ┃ tag             ┃ timestamp  ┃
┡━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ int64  │ int64   │ string          │ int64      │
├────────┼─────────┼─────────────────┼────────────┤
│      2 │   60756 │ funny           │ 1445714994 │
│      2 │   60756 │ Highly quotable │ 1445714996 │
│      2 │   60756 │ will ferrell    │ 1445714992 │
│      2 │   89774 │ Boxing story    │ 1445715207 │
│      2 │   89774 │ MMA             │ 1445715200 │
└────────┴─────────┴─────────────────┴────────────┘

You can join on multiple columns/conditions by passing in a sequence. Show the top 5 users by the number of unique movies that they both rated and tagged:

>>> (
...     tags.join(ratings, ["userId", "movieId"])
...     .group_by(_.userId)
...     .agg(n_rated_and_tagged=_.movieId.nunique())
...     .order_by(_.n_rated_and_tagged.desc())
...     .head(5)
... )
┏━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┓
┃ userId ┃ n_rated_and_tagged ┃
┡━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━┩
│ int64  │ int64              │
├────────┼────────────────────┤
│    474 │               1149 │
│    567 │                109 │
│     62 │                 69 │
│    477 │                 66 │
│    424 │                 58 │
└────────┴────────────────────┘

To self-join a table with itself, you need to call .view() on one of the arguments so the two tables are distinct from each other.

For crafting more complex join conditions, a valid form of a join condition is a 2-tuple like ({left_key}, {right_key}), where each key can be

a Column
Deferred expression
lambda of the form (Table) -> Column

For example, to find all movies pairings that received the same (ignoring case) tags:

>>> movie_tags = tags["movieId", "tag"]
>>> view = movie_tags.view()
>>> movie_tags.join(
...     view,
...     [
...         movie_tags.movieId != view.movieId,
...         (_.tag.lower(), lambda t: t.tag.lower()),
...     ],
... ).head().order_by(("movieId", "movieId_right"))
┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┓
┃ movieId ┃ tag               ┃ movieId_right ┃ tag_right         ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━┩
│ int64   │ string            │ int64         │ string            │
├─────────┼───────────────────┼───────────────┼───────────────────┤
│   60756 │ funny             │          1732 │ funny             │
│   60756 │ Highly quotable   │          1732 │ Highly quotable   │
│   89774 │ Tom Hardy         │        139385 │ tom hardy         │
│  106782 │ drugs             │          1732 │ drugs             │
│  106782 │ Leonardo DiCaprio │          5989 │ Leonardo DiCaprio │
└─────────┴───────────────────┴───────────────┴───────────────────┘

project(resolution=10, columns=('lon', 'lat'), **kwargs)¶

Project the data onto the h3 grid at the given resolution. The column containing the spatial variables to use can be provided as argument in the sptaial_var paramater if it is not (‘lon’,’lat’).

Parameters:

resolution (int)
columns (tuple[str])

Return type:

None

restore()¶: Reloads the original data from the ‘src’ attribute. Usefull to reset the layer after some manipulations have been done and information potentially lost. Overwrites the current state of the layer ?

set_res()¶: Set the reolution of a layer to a given one as target. If no column containing h3_ids is present, project the data to that resolution.

class scalenav.oop.ScalenavProcess¶

__init__(src_db=':memory:', **kwargs)¶

Initiate a duckDB connection with the spatial and H3 extensions.

Return type:: None

create_table(name, obj, **kwargs)¶

Create a table in DuckDB.

Parameters:

name (str) – Name of the table to create
obj (str) – The data with which to populate the table; optional, but at least one of obj or schema must be specified
schema – The schema of the table to create; optional, but at least one of obj or schema must be specified
database –
The name of the database in which to create the table; if not passed, the current database is used.

For multi-level table hierarchies, you can pass in a dotted string path like “catalog.database” or a tuple of strings like (“catalog”, “database”).
temp – Create a temporary table
overwrite – If True, replace the table if it already exists, otherwise fail if the table exists

export()¶

Save the underlying duckDB data base in it’s current state. It contains the table schemas for future use.

Return type:: None

scalenav.oop.add_centr(input)¶

Add the centroid of a hex cell in the table expression.

Parameters:: input (ibis.Table) – A table in the scalenav database.
Returns:: The table with a new column called geom containing centroids of cells from the h3_id column
Return type:: ib.Table

scalenav.oop.change_res(input, levels=1, transform_expr=None)¶

Projecting an H3 indexed table through scales in the DuckDB backend.

Parameters:

input (Table) – a table with an h3_id column containing the index values.
levels (int) – the number of resolutions to change. positive numbers mean increasing, negative decreasing
transform_expr (dict | None) – A dictionary with column names as keys and functions to perform in values. See details for more…

Return type:

Table

Notes

The transformation expression gives great flexibility on the actions to be performed on the columns of the table. It needs to be done in ibis. Refer to the docs for deeper explanation : https://ibis-project.org/reference/expression-generic For negative levels values, the expression needs to perdofm an aggregation. For positive levels, it needs to be a mutation that is applied to the old values of each column.

Returns:

A downscaled ibis table where the columns identified with a _var suffix have been aggregated or disaggregated depending on the direction of the scale change.

Return type:

ib.Table

Parameters:

input (Table)
levels (int)
transform_expr (dict | None)

scalenav.oop.combine(conn, input, name, overwrite=False, fill_val=0)¶

Provide a dict with named tables, the keys will be reused to create variable names in the binded data. Similar effect to rbind in R.

Parameters:

conn (ibis backend connection object, DuckDB recommended) – A connection object
input (dictionary of ibis tables) – A dictionary containing the tables used in the process.
name (str) – The name of the new table that will be created in the backend
overwrite (bool, optional) – overwrite a table with the same name if it already exists, by default False
fill_val (float) – A value to fill the null cells resulting from the outer join

Returns:

An ibis table which is an outer merge of all the provided tables

Return type:

ib.Table

scalenav.oop.concat(input, **kwargs)¶

Stack and regroup a set of layers into 1. Similar effect to concat in pandas.

Parameters:: input (A dict of layers)
Returns:: One combined layer of the inputs stacked
Return type:: Layer

scalenav.oop.connect(database=':memory:', interactive=True, preload_ext=True, community_ext=None, other_ext=None, **kwargs)¶

Create a duckDB connection with spatial and H3 extensions loaded. For details on the additional parameters, refer to : https://duckdb.org/docs/stable/configuration/overview.html

Parameters:

database (str) – a str path to a duckdb data base, or any of [‘memory, ‘:memory:’, ‘mem’] to create a temporary in memory instance in the current session.
interatvie (bool) – to work on in a jupyter notebook
kwargs (dict) – other parameters to the duckDB backend, such as memory limit, nounber of cores - community_ext - other_ext
interactive (bool)
preload_ext (bool)
community_ext (list | None)
other_ext (list | None)

Returns:

A ibis/duckDB connection object.

Return type:

ibis.Table

scalenav.oop.constrain(layer, constraint)¶

UNDER DEV

Parameters:

layer (_type_) – _description_
constraint (_type_) – _description_

scalenav.oop.dump_fill_h3(table, res=10, geometry='geometry', keep=False)¶

Dump and fill with H3 cells

Does not preserve order of values.

Parameters:

table (ibis Table) – An ibis table with a geometry column
res (int, optional) – _description_, by default 10
keep (bool, optional) – Whether to keep the original geometry in the output

Returns:

A table where only the polygonal geometries are preserved, the multipolygons are unnested into individual polygons and then filled with h3 cells. Because of that, the order and original number of rows is not preserved.

Return type:

ibis Table

scalenav.oop.fill_poly(table, res=10, geometry='geometry')¶

Fill a geometry column containing polygons with H3 cells.

Parameters:

table (ibis Table) – A table with polygon geometries.
res (int, optional) – H3 resolution to use, by default 10

Returns:

A table with a new column called ‘h3_id’ containing a list of cell indices for each geometry.

Return type:

ibis Table

scalenav.oop.join(table1, table2, predicates=None, how='outer')¶

Nicer way to outer merge two tables than proposed by ibis. Essentially remove the double columns corresponding to the original predicates.

Parameters:

table1 (_type_) – _description_
table2 (_type_) – _description_
predicates (list, optional) – _description_, by default [“h3_id”]
how (str, optional) – _description_, by default “outer”

Returns:

_description_

Return type:

_type_

scalenav.oop.knn(input, k=1, colname=None)¶

Parameters:

input (ib.Table,Layer]) – _description_
k (int, optional) – _description_, by default 1
colname (_type_, optional) – _description_, by default None

Returns:

_description_

Return type:

_type_

Raises:

IndexError – _description_

scalenav.oop.project(input, res=8, columns=None, keep=True, for_gridding=False)¶

Given an ibis table with coordinates columns, return a table with a new column resulting from generating h3 ids for the points at given h3 resolution.

Parameters:

input (Table) – an ibis table
res (int) – the resolution to project
columns ([<class 'tuple'>, None]) – The columns containing the coordinates if they have unusual names. In order (x,y). Anything like lon, y, easting should be automatically detected.
keep – keep the original coordinates columns in the output
for_gridding (bool)

Returns:

An ibis table

Return type:

ib.Table

scalenav.oop.rast_fill_h3(layer, envelope, res=7, geometry=True, columns=None)¶

Transform a set of points into a h3 voronoi. Usefull for converting rasters where only the centroid is given (ex : outputs of rastapar) into H3 covers. Highly efficient function, requires careful usage as the outputs can get VERY big.

USE AT OWN RISK ! Can be very resource intensive

Parameters:

layer (ibis table) – Typically rastapar output, or the following columns are needed : lon, lat, band_var.
envelope (a shapely box, or polygon) – Limits for the Voronoi process
res (int, optional) – Resolution of cells to fill with, by default 7
geometry (bool, optional) – Whether to return the geometries, by default False
columns (list[str] | None)

scalenav.oop.reduce(table, columns, exclude=True, print_query=False)¶

Aggregate columns in a table on the backend side. Only supports addition of variables at the moment.

Parameters:

table (ibis.Table) – An ibis table.
columns (dict) – A dict matching the desired result to the columns that need aggretgation in the existing table. The key will be used as column name in the return, and the value should be list of columns to add together or a regex matching these columns. Ex : { “KL” : ‘[K-L]’ } will match columns containing K,L and add them into a column named ‘KL’
exclude (bool, optional) – Exclude the original columns from the returned table, by default True
print_query (bool)

Returns:

An ibis table where the desired aggregation has been applied.

Return type:

an ibis Table

scalenav.oop.reindex(input, names_from=None, values_from='id', values_agg='count', values_fill=0)¶

Transpose a variable containing categories associated to located data into an H3 indexed table with new columns based on the categorical values. This performs a pivot. This is essentially a pivot that will be applied to a table in a way to make the h3_id column the index and a categorical variable associated to geolocated features into the columns of the output. Values are aggregated by counting by default, but a function expression can be provided or another common function name as string.

Return type:: A table in which the h3_id column acts as an standard data table index.
Parameters:: input (Table)

scalenav.oop.rescale(input, weight, weight_var='weight_var', weight_geom='geometry', weight_id='id', keep_left=None, keep_right=None)¶

This function takes two layers, assuming the secong parameter contains the layer with data to downscale. Therefore the first layer is supposed to be spatially more granular. It intersects them and rescales the values from the second layer by weighting according to the first layer.

Parameters:

input (Table)
weight (Table)
weight_var (str)
weight_geom (str)
weight_id (str)
keep_left (str | None)
keep_right (str | None)

Return type:

Table

scalenav.oop.simplify(table, geometry='geometry', tolerance=0.1, include=True, preserve_topology=True)¶

Simplify a geometry column in a table. The ibis geospatial method is not registered for a duckdb backend, so this is a workaround.

See : https://duckdb.org/docs/stable/core_extensions/spatial/functions.html#st_simplify

Parameters:

table (ib.Table) – _description_
geometry (str, optional) – _description_, by default “geometry”
tolerance (float, optional) – the tolerance, by default .1
include (bool, optional) – whether to include the old geometry in the output, by default True
preserve_topology (bool, optional) – whether to preserve the original topology, by default True

Returns:

_description_

Return type:

ib.Table

scalenav.oop.table(conn, name, path, coords=None, unnest=True, *args, **kwargs)¶

Create a table in the database out of a parquet file. This function

Parameters:

conn (ibis.backends.duckdb.Backend) – an ibis connection to a duckDB database into which the table should be added
name (str) – a name for the table, good practice is to give the same name as the python variable
path (str) – a path to a parquet file or folder with parquet files with the data
coords (list[str]) – the column names containing the coordinates
kwargs (dict) – some additional arguments can be added, see details
unnest (bool)

Notes

The function support an additional keyword argument at the moment. You can provide a bbox to filter values from the data. The format for the parameter is [xmin,ymin,xmax,ymax]. Another parameter is overwrite to overwirte an existing backend table with the same name.

Returns:

An ibis table object connected to the backend table.

Return type:

ibis.Table

Parameters:

name (str)
path (str)
coords (list[str] | None)
unnest (bool)

scalenav.oop.voronoi(layer, limit=10)¶

UNDER DEV

Parameters:

layer (_type_) – _description_
limit (int, optional) – _description_, by default 10

Returns:

_description_

Return type:

_type_