Evaluation¶

Object-level accuracy assessment of field polygons. See Evaluation for the definitions in one place.

evaluate ¶

Object-level accuracy assessment of field-boundary polygons.

Predicted field polygons are compared with reference field polygons. The functions here report, for a prediction layer and a reference layer:

object detection metrics — precision, recall and F1 of polygon matches at an IoU threshold (one-to-one or legacy many-to-one matching);
area-weighted metrics — the fraction of reference (predicted) area that belongs to matched fields;
segmentation errors — over- and under-segmentation of each reference field against the prediction it overlaps most (Persello & Bruzzone, 2010);
boundary metrics — Hausdorff and symmetric mean boundary distance for matched pairs, boundary precision/recall/F1 within a distance tolerance (buffered-line method) and the fraction of reference fields that are matched with a mean boundary distance within that tolerance;
the same metrics per stratum (a reference attribute) and per field-size class, with percentile bootstrap confidence intervals obtained by resampling reference fields.

:func:evaluate returns the summary dictionary, :func:evaluate_frame the per-reference-field table the summary is built from, and :func:pixels_per_field the number of pixels a field spans at a given ground sample distance.

References

Clinton, N., Holt, A., Scarborough, J., Yan, L., Gong, P., 2010. Accuracy assessment measures for object-based image segmentation goodness. Photogrammetric Engineering & Remote Sensing 76 (3), 289-299. https://doi.org/10.14358/PERS.76.3.289

Persello, C., Bruzzone, L., 2010. A novel protocol for accuracy assessment in classification of very high resolution images. IEEE Transactions on Geoscience and Remote Sensing 48 (3), 1232-1244. https://doi.org/10.1109/TGRS.2009.2029570

Stehman, S.V., Foody, G.M., 2019. Key issues in rigorous accuracy assessment of land cover products. Remote Sensing of Environment 231, 111199. https://doi.org/10.1016/j.rse.2019.05.018

MATCHING_METHODS `module-attribute` ¶

MATCHING_METHODS: tuple[str, ...] = ('one_to_one', 'many_to_one')

BOUNDARY_SAMPLE_SPACING_M `module-attribute` ¶

BOUNDARY_SAMPLE_SPACING_M: float = 1.0

BOUNDARY_MAX_SAMPLES `module-attribute` ¶

BOUNDARY_MAX_SAMPLES: int = 1000

BOOTSTRAP_CONFIDENCE_LEVEL `module-attribute` ¶

BOOTSTRAP_CONFIDENCE_LEVEL: float = 0.95

evaluate ¶

evaluate(predicted: GeoDataFrame | GeoSeries, reference: GeoDataFrame | GeoSeries, iou_threshold: float = 0.5, *, matching: str = 'one_to_one', strata: str | Series | Sequence[Any] | ndarray | None = None, size_bins: str | Sequence[float] | ndarray | None = None, bootstrap: int = 0, bootstrap_seed: int = 42, boundary_tolerance_m: float | None = None, equal_area_crs: Any = None, boundary_sample_spacing_m: float | None = BOUNDARY_SAMPLE_SPACING_M) -> dict[str, Any]

Compute object-level accuracy metrics of predicted field polygons.

Parameters:

Name	Type	Description	Default
`predicted`	`GeoDataFrame or GeoSeries`	Predicted field polygons. Must have a CRS unless it contains no geometries.	required
`reference`	`GeoDataFrame or GeoSeries`	Reference field polygons. Must have a CRS unless it contains no geometries (it may differ from the prediction's CRS).	required
`iou_threshold`	`float`	Minimum IoU (intersection over union) for a prediction and a reference field to match, in `(0, 1]` (default 0.5). A pair matches when `IoU >= iou_threshold` (compared with a relative tolerance of 1e-9 to absorb floating-point rounding) and the intersection has positive area.	`0.5`
`matching`	`('one_to_one', 'many_to_one')`	`"one_to_one"` (default): greedy one-to-one assignment. All pairs with IoU at or above the threshold are sorted by descending IoU (ties by reference then prediction position) and a pair is accepted when neither polygon has been matched yet. This is a greedy assignment in the spirit of COCO-style detection evaluation (which processes detections in order of confidence); without confidence scores, pairs are processed in order of IoU. It is not an optimal (maximum-cardinality) assignment. When neither layer contains overlapping polygons and `iou_threshold > 0.5`, every polygon has at most one partner above the threshold, so the matching is unique. `"many_to_one"`: the 0.1.x behaviour. Each reference field is matched to its highest-IoU prediction at or above the threshold, and one prediction may match several reference fields; `count_tp` counts matched reference fields and `count_fp` counts predictions that matched none, so `precision = TP / (TP + FP)` can exceed the fraction of predictions that are matched.	`"one_to_one"`
`strata`	`(str, Series or array - like)`	Reference stratum of each reference field: the name of a column of reference, a Series aligned with `reference.index`, or an array with one value per reference row. Metrics are also reported per stratum (`per_stratum`). Rows with a missing value contribute to the overall metrics only.	`None`
`size_bins`	`"auto" or sequence of float`	Field-size classes, in hectares (1 ha = 10,000 m²), of the reference field area in equal_area_crs. Either `"auto"` — approximately log-spaced edges from the 1-2-5 series (..., 0.1, 0.2, 0.5, 1, 2, 5, 10, ... ha) spanning the reference areas — or at least two strictly increasing, non-negative edges (the last may be `inf`). Classes are `[lower, upper)` except the last, which is `[lower, upper]` (the :func:`numpy.histogram` convention). Metrics are also reported per class (`per_size_class`); reference fields outside the edges contribute to the overall metrics only.	`None`
`bootstrap`	`int`	Number of bootstrap resamples for confidence intervals (0 disables).	`0`
`bootstrap_seed`	`int`	Seed (a non-negative integer) of the bootstrap random number generator.	`42`
`boundary_tolerance_m`	`float`	Distance tolerance in metres. When given, boundary precision/recall/F1 and coverage within tolerance are computed.	`None`
`equal_area_crs`	`CRS - like`	Projected CRS with axes in metres used for areas and IoU (anything :meth:`pyproj.CRS.from_user_input` accepts). Defaults to :func:`agribound.io.crs.get_equal_area_crs` (EPSG:6933). Areas are preserved only by an equal-area projection, and only on the ellipsoid it is defined for. Spherical projections applied to WGS 84 coordinates, such as ESRI:54009 (Mollweide), change ellipsoidal areas by up to about 0.7 %. Another projected CRS is accepted and used as given, but a WARNING is logged when the area of any evaluated polygon in it differs from its area in EPSG:6933 by more than 0.01 % (relative). The reported areas, area errors, area-weighted metrics and size classes then use the areas in the given CRS.	`None`
`boundary_sample_spacing_m`	`float or None`	Maximum spacing in metres between the sample points of each boundary for the Hausdorff and mean boundary distances. The default, `BOUNDARY_SAMPLE_SPACING_M` (1 m), applies to every boundary, so the sampling error bound (see Notes) is the same for every field whatever its size. `None` selects a length-dependent spacing, `max(BOUNDARY_SAMPLE_SPACING_M, perimeter / BOUNDARY_MAX_SAMPLES)`: 1 m, or perimeter / 1000 for boundaries longer than 1 km. It is faster on layers with many large fields: in our benchmark, the 50,603 NMOSE fields took 33 s instead of 56 s. But its error bound grows with field size, which biases comparisons of boundary distances between size classes. Run time grows with the number of samples.	`BOUNDARY_SAMPLE_SPACING_M`

Returns:

Type Description

dict[str, Any]

Keys kept from 0.1.x (over_segmentation/under_segmentation keep their 0.1.x definitions; the default matching changed to one-to-one):

iou_mean: mean IoU of matched pairs (0.0 when nothing matched).
precision: TP / (TP + FP).
recall: TP / (TP + FN), i.e. matched / evaluated reference fields.
f1: harmonic mean of precision and recall.
over_segmentation: fraction of reference fields with more than one prediction at IoU >= threshold (0.1.x definition).
under_segmentation: fraction of predictions with more than one reference field at IoU >= threshold (0.1.x definition). With non-overlapping polygons and a threshold above 0.5 both are always 0; use oversegmentation_mean and undersegmentation_mean to measure splits and merges.
area_error_mean_m2: mean absolute area difference of matched pairs (0.0 when nothing matched).
count_predicted, count_reference: evaluated polygons.
count_tp, count_fp, count_fn: true positives (matched reference fields), false positives (unmatched predictions) and false negatives (unmatched reference fields).
iou_threshold.

Added in 1.0:

matching, equal_area_crs (e.g. "EPSG:6933"), distance_crs (description) and utm_epsg_codes (UTM zones used for distances).
best_iou_mean: mean over all reference fields of the highest IoU with any prediction (0 for fields no prediction overlaps); unlike iou_mean it is not conditional on a match.
area_weighted_recall: area of matched reference fields / area of all reference fields. area_weighted_precision: area of matched predictions / area of all predictions. area_weighted_f1.
reference_area_ha, predicted_area_ha: total areas.
median_area_ha: median reference field area.
area_error_signed_mean_m2: mean of (predicted - reference) area over matched pairs (NaN when nothing matched).
oversegmentation_mean, undersegmentation_mean: means over reference fields of OS = 1 - |r ∩ p'| / |r| and US = 1 - |r ∩ p'| / |p'|, where p' is the prediction with the largest intersection area with reference field r (ties: the first in prediction order; the maximum-overlap pairing of Persello & Bruzzone, 2010; the measures have the form of Clinton et al.'s (2010) OverSegmentation and UnderSegmentation, which pair each reference object with a different subset of segments). A reference field that no prediction overlaps has OS = 1 and no US (excluded from the US mean). Both are 0 for a perfect delineation.
hausdorff_mean_m, hausdorff_median_m: Hausdorff distance between the boundaries of matched pairs.
boundary_distance_mean_m, boundary_distance_median_m: symmetric mean boundary distance of matched pairs, (∫_∂r d(x, ∂p) dx + ∫_∂p d(x, ∂r) dx) / (|∂r| + |∂p|).
boundary_sample_spacing_m: the argument (None for the length-dependent spacing). boundary_sample_spacing_max_m: the largest spacing Δ used for any matched pair (NaN when nothing matched), which bounds the sampling error of the two distances above (see Notes).
count_fp_unassigned: false positives that overlap no reference field (they cannot be attributed to a stratum or size class). When the reference layer does not contain every field in the area (a registry of some fields only), these predictions may be real fields missing from it; count_tp / (count_tp + count_fp - count_fp_unassigned) is the precision among predictions that overlap a reference field.
count_reference_dropped, count_predicted_dropped: rows excluded because their geometry is null, empty, non-polygonal or has zero area. count_reference_repaired, count_predicted_repaired: evaluated geometries that were invalid and were repaired with shapely.make_valid(method="structure").

With boundary_tolerance_m:

boundary_tolerance_m.
boundary_recall: length of reference boundaries within the tolerance of any predicted boundary / total reference boundary length. boundary_precision: the same for predicted boundaries against reference boundaries. boundary_f1. Lengths are summed per polygon, so an edge shared by two adjacent fields counts once for each (buffered-line method: the length of one layer's boundary inside the union of the tolerance buffers of the other layer's boundaries, computed exactly; see Notes).
coverage_within_tolerance: fraction of reference fields that are matched and whose symmetric mean boundary distance is at most the tolerance; coverage_within_tolerance_area: the same weighted by reference field area.

With strata: strata_column (name or None), count_reference_no_stratum and per_stratum — a dict keyed by str(stratum value) of the core metrics above plus n (reference fields in the stratum). With size_bins: size_class_edges_ha, count_reference_outside_size_classes and per_size_class — keyed "<lower>-<upper>" (ha, formatted with :g, or in full precision if :g would make two keys equal), each with lower_ha/upper_ha. In a group, a prediction counts in the group of its home reference field: its matched reference field if matched (for many-to-one, the matched field with the highest IoU), else the reference field it overlaps most. Predictions overlapping no reference field count only in the overall metrics. A size class without reference fields is reported with n = 0 and NaN rates.

With bootstrap > 0: bootstrap — {"n_resamples", "seed", "confidence_level", "method", "resampling", "ci", "n_undefined"}, where ci maps metric name to [lower, upper]; each per_stratum/per_size_class entry also gains ci (and ci_n_undefined when a metric was undefined in some resamples).

Conventions for empty denominators: the rates (keys ending in precision, recall or f1, iou_mean and the coverage keys) and the 0.1.x keys over_segmentation, under_segmentation and area_error_mean_m2 are 0.0; all other means and distances are NaN. When both layers are empty the rates are 1.0 (0.1.x behaviour).

Raises:

Type	Description
`TypeError`	If an input is not a GeoDataFrame/GeoSeries.
`ValueError`	If an input that contains geometries has no CRS, or an argument is out of range.
`ImportError`	If a layer contains invalid geometries and the installed shapely does not provide `make_valid(method="structure")` (shapely >= 2.1 built with GEOS >= 3.10).

Notes

Geometry handling. Invalid geometries are repaired with shapely.make_valid(method="structure", keep_collapsed=False) and only their polygonal parts are kept; null, empty, non-polygonal and zero-area geometries are excluded (counted in count_*_dropped). The repair needs shapely >= 2.1 built with GEOS >= 3.10. With older versions an ImportError is raised when a repair is needed, rather than switching to the "linework" method, which repairs some geometries differently. A layer without a CRS is accepted only when it has no geometries (for example an empty prediction). Layers that cross the antimeridian are not supported.

Coordinate systems. Areas and IoU are computed in equal_area_crs. Geometries are reprojected vertex by vertex, so each edge becomes a straight chord in the target CRS. Before reprojection to equal_area_crs, edges are split into pieces of at most 50 m in the source CRS. The pieces are collinear there, so the polygons are unchanged in their own CRS. Without this split, a long edge with few vertices would become a single chord, while a coincident edge of the other layer with more vertices follows the curved image of the line, and the two would no longer coincide. For a 2 km × 10 m field that changed the IoU by up to 0.4 % at 34° latitude and 1.5 % at 70°.

With the split, we compared against the planar IoU in the source UTM CRS (inputs in UTM zone 13N). For 100 m and 1 km shifted squares and a notched 2 km × 10 m field, at latitudes from 0.5° to 70°, on and 2.7° off the central meridian, the relative difference was at most 5.4e-6. For 1,500 fields polygonised at 1 m, compared with their own ~20-vertex polygons at 34° N, it was at most 2.5e-7.

Distances (Hausdorff, boundary distances and the tolerance metrics) are computed in the WGS 84 / UTM zone of each polygon's centroid: reference fields and their matched predictions in the reference field's zone, and predicted boundaries in the prediction's zone for boundary precision. Inside a zone the linear scale error is below about 0.1 %. Geometries are not split before reprojection to UTM. From a source CRS in the same zone (for example NAD83 / UTM 13N into WGS 84 / UTM 13N) edges stay straight to within 1e-8 m. A 1 km edge along a parallel that is straight in EPSG:4326 is replaced by a chord up to 1.3 cm from its image at 34° latitude, and 3.4 cm at 60°. The deviation grows with the square of the edge length.

Candidate pairs come from a shapely STRtree query with the "intersects" predicate; intersection areas are computed with vectorised shapely operations.

Boundary distances are computed by sampling each boundary (interior rings included) at points at most Δ apart, keeping every original vertex, and taking the exact distance from each sample to the other boundary. Δ is boundary_sample_spacing_m (1 m by default), or, when that is None, max(BOUNDARY_SAMPLE_SPACING_M, perimeter / BOUNDARY_MAX_SAMPLES): 1 m, or perimeter / 1000 for boundaries longer than 1 km. The distance to a boundary is 1-Lipschitz. So, with Δ the larger spacing of the two boundaries of a pair (boundary_sample_spacing_m in :func:evaluate_frame), the Hausdorff distance is underestimated by at most Δ / 2, and the symmetric mean distance (trapezoidal rule) differs from the exact line-integral mean by at most Δ / 4.

Tolerance metrics are exact up to floating-point rounding. Boundaries are split into their straight segments. For each pair of nearby segments, one from each layer, the part of the first within the tolerance of the second — its intersection with the second's tolerance region, a rectangle capped by two discs — is computed in closed form, and the union of these parts along each segment is measured. This is the length inside the circular-arc buffers of the other layer's boundaries, with no polygonal approximation of the arcs. Run time grows with the number of boundary segments within the tolerance of each other.

Bootstrap. Each resample draws reference fields with replacement (the same number as evaluated); with strata, every stratum — and the set of fields with a missing stratum — is resampled separately at its own size. A resampled field brings its per-field results and the predictions whose home it is; predictions overlapping no reference field are kept unchanged in every resample, so the intervals reflect sampling variability of the reference fields only. Intervals are percentile intervals at BOOTSTRAP_CONFIDENCE_LEVEL (95 %) over the resamples in which the metric is defined; resamples are generated from child seeds of numpy.random.SeedSequence(bootstrap_seed), so the resamples do not depend on how they are batched. Within a resample a rate whose denominator is 0 — for example any metric of a size class none of whose fields was drawn — is undefined rather than 0, and n_undefined counts such resamples. Medians have no intervals. The intervals treat the evaluated reference fields as an independent random sample (a stratified random sample when strata is given). They do not model spatial autocorrelation between neighbouring fields, so they are too narrow when errors are spatially clustered, and they are not design-based estimates for a probability sample of reference fields (see Stehman & Foody, 2019, on sampling design and analysis).

Examples:

>>> from agribound.evaluate import evaluate
>>> metrics = evaluate(predicted_gdf, reference_gdf, boundary_tolerance_m=5.0)
>>> print(f"F1: {metrics['f1']:.3f}, area-weighted recall: "
...       f"{metrics['area_weighted_recall']:.3f}")

Source code in agribound/evaluate.py

def evaluate(
    predicted: gpd.GeoDataFrame | gpd.GeoSeries,
    reference: gpd.GeoDataFrame | gpd.GeoSeries,
    iou_threshold: float = 0.5,
    *,
    matching: str = "one_to_one",
    strata: str | pd.Series | Sequence[Any] | np.ndarray | None = None,
    size_bins: str | Sequence[float] | np.ndarray | None = None,
    bootstrap: int = 0,
    bootstrap_seed: int = 42,
    boundary_tolerance_m: float | None = None,
    equal_area_crs: Any = None,
    boundary_sample_spacing_m: float | None = BOUNDARY_SAMPLE_SPACING_M,
) -> dict[str, Any]:
    """Compute object-level accuracy metrics of predicted field polygons.

    Parameters
    ----------
    predicted : geopandas.GeoDataFrame or geopandas.GeoSeries
        Predicted field polygons. Must have a CRS unless it contains no
        geometries.
    reference : geopandas.GeoDataFrame or geopandas.GeoSeries
        Reference field polygons. Must have a CRS unless it contains no
        geometries (it may differ from the prediction's CRS).
    iou_threshold : float
        Minimum IoU (intersection over union) for a prediction and a reference
        field to match, in ``(0, 1]`` (default 0.5). A pair matches when
        ``IoU >= iou_threshold`` (compared with a relative tolerance of 1e-9 to
        absorb floating-point rounding) and the intersection has positive area.
    matching : {"one_to_one", "many_to_one"}
        ``"one_to_one"`` (default): greedy one-to-one assignment. All pairs
        with IoU at or above the threshold are sorted by descending IoU (ties
        by reference then prediction position) and a pair is accepted when
        neither polygon has been matched yet. This is a greedy assignment in
        the spirit of COCO-style detection evaluation (which processes
        detections in order of confidence); without confidence scores, pairs
        are processed in order of IoU. It is not an optimal (maximum-cardinality)
        assignment. When neither layer contains overlapping polygons and
        ``iou_threshold > 0.5``, every polygon has at most one partner above
        the threshold, so the matching is unique.
        ``"many_to_one"``: the 0.1.x behaviour. Each reference field is matched
        to its highest-IoU prediction at or above the threshold, and one
        prediction may match several reference fields; ``count_tp`` counts
        matched reference fields and ``count_fp`` counts predictions that
        matched none, so ``precision = TP / (TP + FP)`` can exceed the
        fraction of predictions that are matched.
    strata : str, pandas.Series or array-like, optional
        Reference stratum of each reference field: the name of a column of
        *reference*, a Series aligned with ``reference.index``, or an array
        with one value per reference row. Metrics are also reported per
        stratum (``per_stratum``). Rows with a missing value contribute to the
        overall metrics only.
    size_bins : "auto" or sequence of float, optional
        Field-size classes, in **hectares** (1 ha = 10,000 m²), of the
        reference field area in *equal_area_crs*. Either ``"auto"`` —
        approximately log-spaced edges from the 1-2-5 series (..., 0.1, 0.2,
        0.5, 1, 2, 5, 10, ... ha) spanning the reference areas — or at least
        two strictly increasing, non-negative edges (the last may be
        ``inf``). Classes are ``[lower, upper)`` except the last, which is
        ``[lower, upper]`` (the :func:`numpy.histogram` convention). Metrics
        are also reported per class (``per_size_class``); reference fields
        outside the edges contribute to the overall metrics only.
    bootstrap : int
        Number of bootstrap resamples for confidence intervals (0 disables).
    bootstrap_seed : int
        Seed (a non-negative integer) of the bootstrap random number
        generator.
    boundary_tolerance_m : float, optional
        Distance tolerance in metres. When given, boundary precision/recall/F1
        and coverage within tolerance are computed.
    equal_area_crs : CRS-like, optional
        Projected CRS with axes in metres used for areas and IoU (anything
        :meth:`pyproj.CRS.from_user_input` accepts). Defaults to
        :func:`agribound.io.crs.get_equal_area_crs` (EPSG:6933). Areas are
        preserved only by an equal-area projection, and only on the ellipsoid
        it is defined for. Spherical projections applied to WGS 84
        coordinates, such as ESRI:54009 (Mollweide), change ellipsoidal areas
        by up to about 0.7 %. Another projected CRS is accepted and used as
        given, but a WARNING is logged when the area of any evaluated polygon
        in it differs from its area in EPSG:6933 by more than 0.01 %
        (relative). The reported areas, area errors, area-weighted metrics and
        size classes then use the areas in the given CRS.
    boundary_sample_spacing_m : float or None
        Maximum spacing in metres between the sample points of each boundary
        for the Hausdorff and mean boundary distances. The default,
        ``BOUNDARY_SAMPLE_SPACING_M`` (1 m), applies to every boundary, so the
        sampling error bound (see Notes) is the same for every field
        whatever its size. ``None`` selects a length-dependent spacing,
        ``max(BOUNDARY_SAMPLE_SPACING_M, perimeter / BOUNDARY_MAX_SAMPLES)``:
        1 m, or perimeter / 1000 for boundaries longer than 1 km. It is
        faster on layers with many large fields: in our benchmark, the
        50,603 NMOSE fields took 33 s instead of 56 s. But its error bound
        grows with field size, which biases comparisons of boundary distances
        between size classes. Run time grows with the number of samples.

    Returns
    -------
    dict[str, Any]
        Keys kept from 0.1.x (``over_segmentation``/``under_segmentation`` keep
        their 0.1.x definitions; the default matching changed to one-to-one):

        - ``iou_mean``: mean IoU of matched pairs (0.0 when nothing matched).
        - ``precision``: TP / (TP + FP).
        - ``recall``: TP / (TP + FN), i.e. matched / evaluated reference fields.
        - ``f1``: harmonic mean of precision and recall.
        - ``over_segmentation``: fraction of reference fields with more than
          one prediction at IoU >= threshold (0.1.x definition).
        - ``under_segmentation``: fraction of predictions with more than one
          reference field at IoU >= threshold (0.1.x definition). With
          non-overlapping polygons and a threshold above 0.5 both are always
          0; use ``oversegmentation_mean`` and ``undersegmentation_mean`` to
          measure splits and merges.
        - ``area_error_mean_m2``: mean absolute area difference of matched
          pairs (0.0 when nothing matched).
        - ``count_predicted``, ``count_reference``: evaluated polygons.
        - ``count_tp``, ``count_fp``, ``count_fn``: true positives (matched
          reference fields), false positives (unmatched predictions) and
          false negatives (unmatched reference fields).
        - ``iou_threshold``.

        Added in 1.0:

        - ``matching``, ``equal_area_crs`` (e.g. ``"EPSG:6933"``),
          ``distance_crs`` (description) and ``utm_epsg_codes`` (UTM zones
          used for distances).
        - ``best_iou_mean``: mean over all reference fields of the highest IoU
          with any prediction (0 for fields no prediction overlaps); unlike
          ``iou_mean`` it is not conditional on a match.
        - ``area_weighted_recall``: area of matched reference fields / area of
          all reference fields. ``area_weighted_precision``: area of matched
          predictions / area of all predictions. ``area_weighted_f1``.
        - ``reference_area_ha``, ``predicted_area_ha``: total areas.
        - ``median_area_ha``: median reference field area.
        - ``area_error_signed_mean_m2``: mean of (predicted - reference) area
          over matched pairs (NaN when nothing matched).
        - ``oversegmentation_mean``, ``undersegmentation_mean``: means over
          reference fields of ``OS = 1 - |r ∩ p'| / |r|`` and
          ``US = 1 - |r ∩ p'| / |p'|``, where ``p'`` is the prediction with
          the largest intersection area with reference field ``r`` (ties:
          the first in prediction order; the maximum-overlap pairing of
          Persello & Bruzzone, 2010; the measures
          have the form of Clinton et al.'s (2010) OverSegmentation and
          UnderSegmentation, which pair each reference object with a
          different subset of segments). A reference field that no
          prediction overlaps has OS = 1 and no US (excluded from the US
          mean). Both are 0 for a perfect delineation.
        - ``hausdorff_mean_m``, ``hausdorff_median_m``: Hausdorff distance
          between the boundaries of matched pairs.
        - ``boundary_distance_mean_m``, ``boundary_distance_median_m``:
          symmetric mean boundary distance of matched pairs,
          ``(∫_∂r d(x, ∂p) dx + ∫_∂p d(x, ∂r) dx) / (|∂r| + |∂p|)``.
        - ``boundary_sample_spacing_m``: the argument (None for the
          length-dependent spacing). ``boundary_sample_spacing_max_m``: the
          largest spacing ``Δ`` used for any matched pair (NaN when nothing
          matched), which bounds the sampling error of the two distances
          above (see Notes).
        - ``count_fp_unassigned``: false positives that overlap no reference
          field (they cannot be attributed to a stratum or size class). When
          the reference layer does not contain every field in the area (a
          registry of some fields only), these predictions may be real fields
          missing from it; ``count_tp / (count_tp + count_fp -
          count_fp_unassigned)`` is the precision among predictions that
          overlap a reference field.
        - ``count_reference_dropped``, ``count_predicted_dropped``: rows
          excluded because their geometry is null, empty, non-polygonal or
          has zero area. ``count_reference_repaired``,
          ``count_predicted_repaired``: evaluated geometries that were invalid
          and were repaired with ``shapely.make_valid(method="structure")``.

        With ``boundary_tolerance_m``:

        - ``boundary_tolerance_m``.
        - ``boundary_recall``: length of reference boundaries within the
          tolerance of any predicted boundary / total reference boundary
          length. ``boundary_precision``: the same for predicted boundaries
          against reference boundaries. ``boundary_f1``. Lengths are summed
          per polygon, so an edge shared by two adjacent fields counts once
          for each (buffered-line method: the length of one layer's boundary
          inside the union of the tolerance buffers of the other layer's
          boundaries, computed exactly; see Notes).
        - ``coverage_within_tolerance``: fraction of reference fields that are
          matched **and** whose symmetric mean boundary distance is at most the
          tolerance; ``coverage_within_tolerance_area``: the same weighted by
          reference field area.

        With ``strata``: ``strata_column`` (name or None),
        ``count_reference_no_stratum`` and ``per_stratum`` — a dict keyed by
        ``str(stratum value)`` of the core metrics above plus ``n`` (reference
        fields in the stratum). With ``size_bins``: ``size_class_edges_ha``,
        ``count_reference_outside_size_classes`` and ``per_size_class`` —
        keyed ``"<lower>-<upper>"`` (ha, formatted with ``:g``, or in full
        precision if ``:g`` would make two keys equal), each with
        ``lower_ha``/``upper_ha``.
        In a group, a prediction counts in the group of its *home* reference
        field: its matched reference field if matched (for many-to-one, the
        matched field with the highest IoU), else the reference field it
        overlaps most. Predictions overlapping no reference field count only
        in the overall metrics. A size class without reference fields is
        reported with ``n = 0`` and NaN rates.

        With ``bootstrap > 0``: ``bootstrap`` — ``{"n_resamples", "seed",
        "confidence_level", "method", "resampling", "ci", "n_undefined"}``,
        where ``ci`` maps metric name to ``[lower, upper]``; each
        ``per_stratum``/``per_size_class`` entry also gains ``ci`` (and
        ``ci_n_undefined`` when a metric was undefined in some resamples).

        Conventions for empty denominators: the rates (keys ending in
        ``precision``, ``recall`` or ``f1``, ``iou_mean`` and the coverage
        keys) and the 0.1.x keys ``over_segmentation``, ``under_segmentation``
        and ``area_error_mean_m2`` are 0.0; all other means and distances are
        NaN. When both layers are empty the rates are 1.0 (0.1.x behaviour).

    Raises
    ------
    TypeError
        If an input is not a GeoDataFrame/GeoSeries.
    ValueError
        If an input that contains geometries has no CRS, or an argument is out
        of range.
    ImportError
        If a layer contains invalid geometries and the installed shapely does
        not provide ``make_valid(method="structure")`` (shapely >= 2.1 built
        with GEOS >= 3.10).

    Notes
    -----
    **Geometry handling.** Invalid geometries are repaired with
    ``shapely.make_valid(method="structure", keep_collapsed=False)`` and only
    their polygonal parts are kept; null, empty, non-polygonal and zero-area
    geometries are excluded (counted in ``count_*_dropped``). The repair
    needs shapely >= 2.1 built with GEOS >= 3.10. With older versions an
    ImportError is raised when a repair is needed, rather than switching to
    the ``"linework"`` method, which repairs some geometries differently. A layer
    without a CRS is accepted only when it has no geometries (for example an
    empty prediction). Layers that cross the antimeridian are not supported.

    **Coordinate systems.** Areas and IoU are computed in *equal_area_crs*.
    Geometries are reprojected vertex by vertex, so each edge becomes a
    straight chord in the target CRS. Before reprojection to *equal_area_crs*,
    edges are split into pieces of at most 50 m in the source CRS. The pieces
    are collinear there, so the polygons are unchanged in their own CRS.
    Without this split, a long edge with few vertices would become a single
    chord, while a coincident edge of the other layer with more vertices
    follows the curved image of the line, and the two would no longer
    coincide. For a 2 km × 10 m field that changed the IoU by up to 0.4 % at
    34° latitude and 1.5 % at 70°.

    With the split, we compared against the planar IoU in the source UTM
    CRS (inputs in UTM zone 13N). For 100 m and 1 km shifted squares and a
    notched 2 km × 10 m field, at latitudes from 0.5° to 70°, on and 2.7° off
    the central meridian, the relative difference was at most 5.4e-6. For
    1,500 fields polygonised at 1 m, compared with their own ~20-vertex
    polygons at 34° N, it was at most 2.5e-7.

    Distances (Hausdorff, boundary distances and the tolerance metrics) are
    computed in the WGS 84 / UTM zone of each polygon's centroid: reference
    fields and their matched predictions in the reference field's zone, and
    predicted boundaries in the prediction's zone for boundary precision.
    Inside a zone the linear scale error is below about 0.1 %. Geometries are
    not split before reprojection to UTM. From a source CRS in the same zone
    (for example NAD83 / UTM 13N into WGS 84 / UTM 13N) edges stay straight to
    within 1e-8 m. A 1 km edge along a parallel that is straight in EPSG:4326
    is replaced by a chord up to 1.3 cm from its image at 34° latitude, and
    3.4 cm at 60°. The deviation grows with the square of the edge length.

    **Candidate pairs** come from a shapely ``STRtree`` query with the
    ``"intersects"`` predicate; intersection areas are computed with
    vectorised shapely operations.

    **Boundary distances** are computed by sampling each boundary (interior
    rings included) at points at most ``Δ`` apart, keeping every original
    vertex, and taking the exact distance from each sample to the other
    boundary. ``Δ`` is *boundary_sample_spacing_m* (1 m by default), or,
    when that is None, ``max(BOUNDARY_SAMPLE_SPACING_M, perimeter /
    BOUNDARY_MAX_SAMPLES)``: 1 m, or perimeter / 1000 for boundaries longer
    than 1 km. The distance to a boundary is 1-Lipschitz. So, with ``Δ`` the
    larger spacing of the two boundaries of a pair
    (``boundary_sample_spacing_m`` in :func:`evaluate_frame`), the Hausdorff
    distance is underestimated by at most ``Δ / 2``, and the symmetric mean
    distance (trapezoidal rule) differs from the exact line-integral mean by
    at most ``Δ / 4``.

    **Tolerance metrics** are exact up to floating-point rounding. Boundaries
    are split into their straight segments. For each pair of nearby segments,
    one from each layer, the part of the first within the tolerance of the
    second — its intersection with the second's tolerance region, a
    rectangle capped by two discs — is computed in closed form, and the
    union of these parts along each segment is measured. This is the length
    inside the circular-arc buffers of the other layer's boundaries, with no
    polygonal approximation of the arcs. Run time grows with the number of
    boundary segments within the tolerance of each other.

    **Bootstrap.** Each resample draws reference fields with replacement (the
    same number as evaluated); with *strata*, every stratum — and the set of
    fields with a missing stratum — is resampled separately at its own size.
    A resampled field brings its per-field results and the predictions whose
    home it is; predictions overlapping no reference field are kept unchanged
    in every resample, so the intervals reflect sampling variability of the
    reference fields only. Intervals are percentile intervals at
    ``BOOTSTRAP_CONFIDENCE_LEVEL`` (95 %) over the resamples in which the
    metric is defined; resamples are generated from child seeds of
    ``numpy.random.SeedSequence(bootstrap_seed)``, so the resamples do not
    depend on how they are batched. Within a resample a rate whose
    denominator is 0 — for example any metric of a size class none of whose
    fields was drawn — is undefined rather than 0, and ``n_undefined`` counts
    such resamples. Medians have no intervals. The intervals treat the
    evaluated reference fields as an independent random sample (a stratified
    random sample when *strata* is given). They do not model spatial
    autocorrelation between neighbouring fields, so they are too narrow when
    errors are spatially clustered, and they are not design-based estimates
    for a probability sample of reference fields (see Stehman & Foody, 2019,
    on sampling design and analysis).

    Examples
    --------
    >>> from agribound.evaluate import evaluate
    >>> metrics = evaluate(predicted_gdf, reference_gdf, boundary_tolerance_m=5.0)
    >>> print(f"F1: {metrics['f1']:.3f}, area-weighted recall: "
    ...       f"{metrics['area_weighted_recall']:.3f}")
    """
    params = _Params.from_args(
        iou_threshold=iou_threshold,
        matching=matching,
        bootstrap=bootstrap,
        bootstrap_seed=bootstrap_seed,
        boundary_tolerance_m=boundary_tolerance_m,
        equal_area_crs=equal_area_crs,
        boundary_sample_spacing_m=boundary_sample_spacing_m,
    )
    run = _run(predicted, reference, params, strata=strata, size_bins=size_bins)
    metrics = _summarise(run)
    if params.bootstrap > 0:
        _add_bootstrap(metrics, run)

    logger.info(
        "Evaluation (%s, IoU>=%.2f): P=%.3f R=%.3f F1=%.3f IoU=%.3f "
        "area-weighted R=%.3f (TP=%d FP=%d FN=%d)",
        params.matching,
        params.iou_threshold,
        metrics["precision"],
        metrics["recall"],
        metrics["f1"],
        metrics["iou_mean"],
        metrics["area_weighted_recall"],
        metrics["count_tp"],
        metrics["count_fp"],
        metrics["count_fn"],
    )
    return metrics

evaluate_frame ¶

evaluate_frame(predicted: GeoDataFrame | GeoSeries, reference: GeoDataFrame | GeoSeries, *, iou_threshold: float = 0.5, matching: str = 'one_to_one', strata: str | Series | Sequence[Any] | ndarray | None = None, equal_area_crs: Any = None, boundary_tolerance_m: float | None = None, size_bins: str | Sequence[float] | ndarray | None = None, boundary_sample_spacing_m: float | None = BOUNDARY_SAMPLE_SPACING_M) -> pd.DataFrame

Return the per-reference-field evaluation table.

One row per evaluated reference field (null, empty, non-polygonal and zero-area reference geometries are excluded), indexed by the reference index, so the table can be joined back to reference. The arguments and methods are those of :func:evaluate; the summary returned by :func:evaluate is computed from the same per-field results.

Parameters:

Name	Type	Description	Default
`predicted`	`GeoDataFrame \| GeoSeries`	See :func:`evaluate`.	required
`reference`	`GeoDataFrame \| GeoSeries`	See :func:`evaluate`.	required
`iou_threshold`	`GeoDataFrame \| GeoSeries`	See :func:`evaluate`.	required
`matching`	`GeoDataFrame \| GeoSeries`	See :func:`evaluate`.	required
`strata`	`GeoDataFrame \| GeoSeries`	See :func:`evaluate`.	required
`equal_area_crs`	`GeoDataFrame \| GeoSeries`	See :func:`evaluate`.	required
`boundary_tolerance_m`	`float`	See :func:`evaluate`; adds the tolerance columns.	`None`
`size_bins`	`"auto" or sequence of float`	See :func:`evaluate` (hectares); adds a `size_class` column.	`None`
`boundary_sample_spacing_m`	`float or None`	See :func:`evaluate` (default 1 m).	`BOUNDARY_SAMPLE_SPACING_M`

Returns:

Type Description

DataFrame

Columns:

area_m2, area_ha: reference field area in equal_area_crs.
perimeter_m: reference boundary length (UTM zone of the field).
utm_epsg: EPSG code of that UTM zone.
stratum (with strata), size_class (with size_bins; None outside the edges).
matched: bool. pred_index: index label of the matched prediction (None when unmatched). iou: IoU with it (NaN when unmatched). pred_area_m2: its area (NaN when unmatched).
best_iou: highest IoU with any prediction (0 when none overlaps). n_overlapping_pred: predictions with a positive-area intersection.
max_overlap_pred_index: index label of the prediction with the largest intersection (ties: the first in prediction order; None when none overlaps); oversegmentation, undersegmentation: OS and US against it (see :func:evaluate; US is NaN when no prediction overlaps).
hausdorff_m, boundary_mean_distance_m: boundary distances to the matched prediction (NaN when unmatched). boundary_sample_spacing_m: the sample spacing Δ behind them (the larger of the two boundaries' spacings; NaN when unmatched). The Hausdorff distance is underestimated by at most Δ / 2, and the mean distance is within Δ / 4 of its exact value (see :func:evaluate, Notes).
With boundary_tolerance_m: boundary_within_tolerance_m (length of the field boundary within the tolerance of any predicted boundary), boundary_recall (that length / perimeter_m) and within_tolerance (matched and boundary_mean_distance_m at most the tolerance).

DataFrame.attrs records iou_threshold, matching, equal_area_crs, boundary_tolerance_m and boundary_sample_spacing_m.

Source code in agribound/evaluate.py

def evaluate_frame(
    predicted: gpd.GeoDataFrame | gpd.GeoSeries,
    reference: gpd.GeoDataFrame | gpd.GeoSeries,
    *,
    iou_threshold: float = 0.5,
    matching: str = "one_to_one",
    strata: str | pd.Series | Sequence[Any] | np.ndarray | None = None,
    equal_area_crs: Any = None,
    boundary_tolerance_m: float | None = None,
    size_bins: str | Sequence[float] | np.ndarray | None = None,
    boundary_sample_spacing_m: float | None = BOUNDARY_SAMPLE_SPACING_M,
) -> pd.DataFrame:
    """Return the per-reference-field evaluation table.

    One row per evaluated reference field (null, empty, non-polygonal and
    zero-area reference geometries are excluded), indexed by the reference
    index, so the table can be joined back to *reference*. The arguments and
    methods are those of :func:`evaluate`; the summary returned by
    :func:`evaluate` is computed from the same per-field results.

    Parameters
    ----------
    predicted, reference, iou_threshold, matching, strata, equal_area_crs
        See :func:`evaluate`.
    boundary_tolerance_m : float, optional
        See :func:`evaluate`; adds the tolerance columns.
    size_bins : "auto" or sequence of float, optional
        See :func:`evaluate` (hectares); adds a ``size_class`` column.
    boundary_sample_spacing_m : float or None
        See :func:`evaluate` (default 1 m).

    Returns
    -------
    pandas.DataFrame
        Columns:

        - ``area_m2``, ``area_ha``: reference field area in *equal_area_crs*.
        - ``perimeter_m``: reference boundary length (UTM zone of the field).
        - ``utm_epsg``: EPSG code of that UTM zone.
        - ``stratum`` (with *strata*), ``size_class`` (with *size_bins*;
          None outside the edges).
        - ``matched``: bool. ``pred_index``: index label of the matched
          prediction (None when unmatched). ``iou``: IoU with it (NaN when
          unmatched). ``pred_area_m2``: its area (NaN when unmatched).
        - ``best_iou``: highest IoU with any prediction (0 when none
          overlaps). ``n_overlapping_pred``: predictions with a positive-area
          intersection.
        - ``max_overlap_pred_index``: index label of the prediction with the
          largest intersection (ties: the first in prediction order; None
          when none overlaps);
          ``oversegmentation``, ``undersegmentation``: OS and US against it
          (see :func:`evaluate`; US is NaN when no prediction overlaps).
        - ``hausdorff_m``, ``boundary_mean_distance_m``: boundary distances to
          the matched prediction (NaN when unmatched).
          ``boundary_sample_spacing_m``: the sample spacing ``Δ`` behind them
          (the larger of the two boundaries' spacings; NaN when unmatched).
          The Hausdorff distance is underestimated by at most ``Δ / 2``, and
          the mean distance is within ``Δ / 4`` of its exact value (see
          :func:`evaluate`, Notes).
        - With *boundary_tolerance_m*: ``boundary_within_tolerance_m`` (length
          of the field boundary within the tolerance of any predicted
          boundary), ``boundary_recall`` (that length / ``perimeter_m``) and
          ``within_tolerance`` (matched and ``boundary_mean_distance_m`` at
          most the tolerance).

        ``DataFrame.attrs`` records ``iou_threshold``, ``matching``,
        ``equal_area_crs``, ``boundary_tolerance_m`` and
        ``boundary_sample_spacing_m``.
    """
    params = _Params.from_args(
        iou_threshold=iou_threshold,
        matching=matching,
        bootstrap=0,
        bootstrap_seed=0,
        boundary_tolerance_m=boundary_tolerance_m,
        equal_area_crs=equal_area_crs,
        boundary_sample_spacing_m=boundary_sample_spacing_m,
    )
    run = _run(predicted, reference, params, strata=strata, size_bins=size_bins)
    return _frame(run)

pixels_per_field ¶

pixels_per_field(reference: GeoDataFrame | GeoSeries, gsd_m: float, equal_area_crs: Any = None) -> pd.Series

Return the number of pixels each field spans at a ground sample distance.

p = A / gsd_m², with A the field area in m² in equal_area_crs: the number of gsd_m × gsd_m pixels whose total area equals the field area (not a count of pixel centres inside the field).

Parameters:

Name	Type	Description	Default
`reference`	`GeoDataFrame or GeoSeries`	Field polygons. Must have a CRS unless it contains no geometries (then every value is NaN).	required
`gsd_m`	`float`	Ground sample distance (pixel size) in metres, > 0.	required
`equal_area_crs`	`CRS - like`	Projected CRS for areas (default EPSG:6933). As in :func:`evaluate`, a WARNING is logged when it does not preserve the fields' areas.	`None`

Returns:

Type	Description
`Series`	Float series named `"pixels_per_field"` with the index of reference. Invalid geometries are repaired as in :func:`evaluate`; null, empty, non-polygonal and zero-area geometries give NaN.

Raises:

Type	Description
`TypeError`	If reference is not a GeoDataFrame/GeoSeries.
`ValueError`	If gsd_m is not a positive finite number, equal_area_crs is not a projected CRS in metres, or reference contains geometries but has no CRS.
`ImportError`	If invalid geometries need repair and the installed shapely does not provide `make_valid(method="structure")` (see :func:`evaluate`).

Source code in agribound/evaluate.py

def pixels_per_field(
    reference: gpd.GeoDataFrame | gpd.GeoSeries,
    gsd_m: float,
    equal_area_crs: Any = None,
) -> pd.Series:
    """Return the number of pixels each field spans at a ground sample distance.

    ``p = A / gsd_m²``, with ``A`` the field area in m² in *equal_area_crs*:
    the number of ``gsd_m × gsd_m`` pixels whose total area equals the field
    area (not a count of pixel centres inside the field).

    Parameters
    ----------
    reference : geopandas.GeoDataFrame or geopandas.GeoSeries
        Field polygons. Must have a CRS unless it contains no geometries
        (then every value is NaN).
    gsd_m : float
        Ground sample distance (pixel size) in metres, > 0.
    equal_area_crs : CRS-like, optional
        Projected CRS for areas (default EPSG:6933). As in :func:`evaluate`,
        a WARNING is logged when it does not preserve the fields' areas.

    Returns
    -------
    pandas.Series
        Float series named ``"pixels_per_field"`` with the index of
        *reference*. Invalid geometries are repaired as in :func:`evaluate`;
        null, empty, non-polygonal and zero-area geometries give NaN.

    Raises
    ------
    TypeError
        If *reference* is not a GeoDataFrame/GeoSeries.
    ValueError
        If *gsd_m* is not a positive finite number, *equal_area_crs* is not a
        projected CRS in metres, or *reference* contains geometries but has
        no CRS.
    ImportError
        If invalid geometries need repair and the installed shapely does not
        provide ``make_valid(method="structure")`` (see :func:`evaluate`).
    """
    gsd = _positive_float(gsd_m, "gsd_m")
    ea_crs = _resolve_equal_area_crs(equal_area_crs)
    layer = _prepare_layer(reference, "reference", ea_crs)
    _check_equal_area(ea_crs, (layer,))
    values = np.full(layer.n_input, np.nan)
    values[layer.keep] = layer.area / (gsd * gsd)
    return pd.Series(values, index=layer.index, name="pixels_per_field")

Evaluation¶

evaluate ¶

MATCHING_METHODS module-attribute ¶

BOUNDARY_SAMPLE_SPACING_M module-attribute ¶

BOUNDARY_MAX_SAMPLES module-attribute ¶

BOOTSTRAP_CONFIDENCE_LEVEL module-attribute ¶

evaluate ¶

evaluate_frame ¶

pixels_per_field ¶

MATCHING_METHODS `module-attribute` ¶

BOUNDARY_SAMPLE_SPACING_M `module-attribute` ¶

BOUNDARY_MAX_SAMPLES `module-attribute` ¶

BOOTSTRAP_CONFIDENCE_LEVEL `module-attribute` ¶