Local Data Module¶

local ¶

Local precipitation data sources for pyCropWat.

This module lets you drive pycropwat.EffectivePrecipitation with your own precipitation rasters instead of downloading precipitation from Google Earth Engine. Two on-disk layouts are supported:

Raster mode - a directory (or glob) of one GeoTIFF per month, where the file name carries the year and month, e.g. Precip_2005_07.tif.
NetCDF mode - one or more NetCDF/HDF5 files holding a precipitation variable with a time coordinate (or a single month per file, dated by file name).

Precipitation is returned as a 2-D xarray.DataArray with dims ('y', 'x'), float32 values in millimetres, nodata as NaN, a north-up y coordinate and a CRS attached via rioxarray. That is exactly what the rest of pyCropWat expects, so the precipitation-only methods (cropwat, fao_aglw, fixed_percentage, dependable_rainfall, farmwest) run with no Earth Engine involvement at all.

Classes:

Name	Description
`LocalPrecipitationSource`	Indexed, lazily-opened reader for local monthly precipitation data.

Functions:

Name	Description
`parse_year_month`	Extract `(year, month)` from a file name or arbitrary string.
`open_local_precipitation`	Convenience factory returning a `LocalPrecipitationSource`.

Example

from pycropwat.local import LocalPrecipitationSource

# A directory of monthly WRF GeoTIFFs named Precip_YYYY_MM.tif
src = LocalPrecipitationSource(
    '../pyCropWat_Data/Precip',
    pattern='Precip_*.tif',
    nodata=-9999,
)
print(src.kind, len(src), src.year_range, src.crs, src.shape)

da = src.get_month(2005, 7)     # xr.DataArray, dims ('y', 'x'), mm
print(float(da.mean()))

NetCDF, with an explicit variable and a unit conversion (m -> mm):

from pycropwat.local import open_local_precipitation

with open_local_precipitation('precip_2000_2020.nc',
                              variable='tp',
                              scale_factor=1000.0) as src:
    for year, month in src.available_months():
        da = src.get_month(year, month)

LocalPrecipitationSource ¶

LocalPrecipitationSource(path: Union[str, Path], pattern: str = '*.tif', variable: Optional[str] = None, scale_factor: float = 1.0, nodata: Optional[Union[float, Sequence[float]]] = None, crs: Optional[Union[str, CRS]] = None, date_regex: Optional[str] = None, time_dim: str = 'time', x_dim: Optional[str] = None, y_dim: Optional[str] = None)

Reader for user-supplied local monthly precipitation data.

Wraps a directory of single-month rasters, or one/many NetCDF files, and exposes them as analysis-ready monthly xarray.DataArray objects. Files are indexed up front (cheap) and pixel data is only read when get_month is called.

Parameters:

Name	Type	Description	Default
`path`	`str or Path`	A directory of rasters, a single raster/NetCDF file, or a glob string (e.g. `'/data/precip/Precip_*.tif'`).	required
`pattern`	`str`	Glob applied when `path` is a directory. Files whose suffix is not a recognised raster/NetCDF suffix are ignored, so stray companions such as `.aux.xml`, `metadata.csv` or `.geeup-state.json` never break indexing.	`'*.tif'`
`variable`	`str`	NetCDF variable holding precipitation. Auto-detected when None.	`None`
`scale_factor`	`float`	Constant multiplier applied to every returned array, for converting the source units to millimetres per month: `1000` for metres, `10` for centimetres, `25.4` for inches, `0.1` for tenths of a millimetre. A single multiplier cannot convert a rate such as kg m-2 s-1 to a monthly total, because the factor depends on the number of days in each month - accumulate such data to monthly totals before handing it to pyCropWat.	`1.0`
`nodata`	`float or sequence of float`	Extra nodata sentinel(s) to mask, in addition to the nodata declared in the file metadata. Compared with `numpy.isclose` and NaN-safe, e.g. `-9999`.	`None`
`crs`	`str or CRS`	CRS to attach to the returned arrays (e.g. `'EPSG:4326'`). It overrides whatever the file declares, and logs at INFO when it replaces a declared CRS. Overriding relabels the grid only: the pixels and their coordinates are used exactly as they are on disk and nothing is reprojected, so this is the fix for missing or wrong CRS metadata, not a way to change projection. When None, the file's own CRS is used, falling back to EPSG:4326 with a warning. Passing it does not switch off the mixed-CRS check: raster mode still compares the files' declared CRSs with each other, and reports a directory whose files disagree - see the grid note below.	`None`
`date_regex`	`str`	Custom regex with named groups `year` and `month` used to date file names. See `parse_year_month`.	`None`
`time_dim`	`str`	Name of the NetCDF time coordinate. Falls back to `TIME_DIM_NAMES` when the given name is absent.	`'time'`
`x_dim`	`str`	Name of the NetCDF X dimension. Auto-detected from `X_DIM_NAMES` when None.	`None`
`y_dim`	`str`	Name of the NetCDF Y dimension. Auto-detected from `Y_DIM_NAMES` when None.	`None`

Attributes:

Name	Type	Description
`path`	`Path`	The path/glob the source was constructed from.
`scale_factor`	`float`	Multiplier applied to returned data.

Raises:

Type	Description
`FileNotFoundError`	If `path` does not exist and matches nothing.
`ValueError`	If no usable files are found, raster and NetCDF files are mixed, no file name could be dated, or the NetCDF precipitation variable is ambiguous.

Notes

Raster layout limitation. Raster mode expects one file per month, dated by file name. A single multi-band raster per year (12 bands = 12 months) is not supported and raises a ValueError: split it into monthly files, or supply a date_regex matching a one-file-per-month naming scheme. A raster that is dated to a single month but happens to carry extra bands is accepted - the first band is used and a warning is logged.

Every array returned by get_month has dims exactly ('y', 'x'), dtype float32, a descending (north-up) y coordinate, nodata as NaN, the source CRS written via rioxarray, and attrs {'units', 'long_name', 'year', 'month', 'source'}.

One grid for every month. All months are stacked into a single time series downstream, so every file must share one CRS, shape and affine transform. Raster mode checks that up front from file metadata only (no pixels are read) and raises a ValueError naming the first offending file and how it differs; NetCDF mode catches a mismatched grid when the month is read. That check always compares the CRSs the files declare with each other, never with the crs argument, so a mixed-CRS directory is caught whether or not an override was supplied: without crs it is one of the grid differences that raise, and with crs the override still wins (that is what it is for) but the disagreement is logged as a WARNING, since relabelling never reprojects. Hence shape, resolution, bounds and native_bounds describe every month, and bounds / native_bounds are always ordered (minx, miny, maxx, maxy), including for south-up files.

Curvilinear NetCDF coordinates. A file with 2-D coordinates (the WRF XLAT / XLONG layout on south_north / west_east dims) is supported, whether xarray has promoted them to coordinates (the file gives the precipitation variable a CF coordinates = "XLONG XLAT" attribute) or they sit in the file as plain 2-D data variables on the same dims, as hand-cut and post-processed subsets often leave them. Either way the 2-D coordinates are collapsed to 1-D along the middle row/column. Map-projected model output is always mildly curvilinear in lon/lat, so the resulting displacement is measured - beyond half a cell it is logged as a WARNING quantifying the error, and beyond 20% of the axis span (or if the collapsed coordinate is not monotonic) it raises rather than silently returning a wrong grid. A dimension with no usable coordinate at all falls back to pixel indices with a WARNING: the shape survives but the grid is not georeferenced.

Examples:

Directory of monthly GeoTIFFs:

from pycropwat.local import LocalPrecipitationSource

src = LocalPrecipitationSource('../pyCropWat_Data/Precip',
                               pattern='Precip_*.tif',
                               nodata=-9999)
print(src.kind, len(src), src.year_range)     # raster 264 (2000, 2021)
da = src.get_month(2005, 7)
print(da.dims, da.shape)                      # ('y', 'x') (689, 799)
src.close()

NetCDF stack used as a context manager:

with LocalPrecipitationSource('precip.nc', variable='precip') as src:
    print(src.available_months()[:3])
    da = src.get_month(2005, 7)

Source code in pycropwat/local.py

def __init__(
    self,
    path: Union[str, Path],
    pattern: str = '*.tif',
    variable: Optional[str] = None,
    scale_factor: float = 1.0,
    nodata: Optional[Union[float, Sequence[float]]] = None,
    crs: Optional[Union[str, CRS]] = None,
    date_regex: Optional[str] = None,
    time_dim: str = 'time',
    x_dim: Optional[str] = None,
    y_dim: Optional[str] = None
):
    self.path = Path(str(path))
    self.pattern = pattern
    self.scale_factor = float(scale_factor)

    self._variable = variable
    self._crs_arg = crs
    self._date_regex = date_regex
    self._time_dim = time_dim
    self._x_dim = x_dim
    self._y_dim = y_dim

    if nodata is None:
        self._nodata_values = ()
    elif isinstance(nodata, (list, tuple, set, np.ndarray)):
        self._nodata_values = tuple(float(value) for value in nodata)
    else:
        self._nodata_values = (float(nodata),)

    # Populated by the mode-specific setup below.
    self._files = self._resolve_files(path, pattern)
    self._kind = self._resolve_kind(self._files)

    self._index: Dict[Tuple[int, int], Any] = {}
    self._dataset: Optional[xr.Dataset] = None
    self._data_array: Optional[xr.DataArray] = None
    self._file_cache: Dict[Path, xr.Dataset] = {}
    self._time_name: Optional[str] = None
    self._var_name: Optional[str] = None
    self._crs: Optional[CRS] = None
    self._shape: Optional[Tuple[int, int]] = None
    self._resolution: Optional[Tuple[float, float]] = None
    self._native_bounds: Optional[Tuple[float, float, float, float]] = None
    self._bounds: Optional[Tuple[float, float, float, float]] = None

    if self._kind == 'raster':
        self._setup_raster()
    else:
        self._setup_netcdf()

    if not self._index:
        raise ValueError(
            f"No dated months could be resolved from {self.path}. "
            f"Expected file names such as 'Precip_2005_07.tif' or a NetCDF time "
            f"coordinate. Pass date_regex=r'(?P<year>\\d{{4}})_(?P<month>\\d{{2}})' "
            f"(adapted to your naming) to override the built-in parsing."
        )

    self._compute_bounds()

    years = self.year_range
    logger.info(
        "Local precipitation source: kind=%s, files=%d, months=%d, years=%d-%d, "
        "crs=%s, shape=%s",
        self._kind, len(self._files), len(self._index), years[0], years[1],
        self._crs, self._shape
    )

bounds `property` ¶

bounds: Tuple[float, float, float, float]

tuple : (minx, miny, maxx, maxy) in EPSG:4326 lon/lat degrees.

crs `property` ¶

crs: CRS

CRS : Coordinate reference system of the source data.

files `property` ¶

files: List[Path]

list of Path : The data files backing this source, sorted.

kind `property` ¶

kind: str

str : 'raster' for one-file-per-month rasters, 'netcdf' otherwise.

native_bounds `property` ¶

native_bounds: Tuple[float, float, float, float]

tuple : (minx, miny, maxx, maxy) in the source CRS.

resolution `property` ¶

resolution: Tuple[float, float]

tuple : (x_resolution, y_resolution) as positive values in CRS units.

shape `property` ¶

shape: Tuple[int, int]

tuple : (n_rows, n_cols) of every monthly grid.

In raster mode this is validated across all indexed files when the source is built; a file on a different grid raises rather than being read.

year_range `property` ¶

year_range: Tuple[int, int]

tuple : (first_year, last_year) present in the source.

enter ¶

__enter__() -> LocalPrecipitationSource

Return self so the source can be used in a with block.

Source code in pycropwat/local.py

def __enter__(self) -> "LocalPrecipitationSource":
    """Return self so the source can be used in a ``with`` block."""
    return self

exit ¶

__exit__(exc_type, exc_value, traceback) -> bool

Close open file handles on exit; never suppresses exceptions.

Source code in pycropwat/local.py

def __exit__(self, exc_type, exc_value, traceback) -> bool:
    """Close open file handles on exit; never suppresses exceptions."""
    self.close()
    return False

len ¶

__len__() -> int

int : Number of months available in this source.

Source code in pycropwat/local.py

def __len__(self) -> int:
    """int : Number of months available in this source."""
    return len(self._index)

repr ¶

__repr__() -> str

Return a concise, informative representation.

Source code in pycropwat/local.py

def __repr__(self) -> str:
    """Return a concise, informative representation."""
    years = self.year_range
    return (
        f"LocalPrecipitationSource(kind='{self._kind}', files={len(self._files)}, "
        f"months={len(self._index)}, years={years[0]}-{years[1]}, "
        f"crs='{self._crs}', shape={self._shape})"
    )

available_months ¶

available_months() -> List[Tuple[int, int]]

List every month present in the source.

Returns:

Type	Description
`list`	Sorted list of `(year, month)` tuples.

Examples:

src.available_months()[:2]   # [(2000, 1), (2000, 2)]

Source code in pycropwat/local.py

def available_months(self) -> List[Tuple[int, int]]:
    """
    List every month present in the source.

    Returns
    -------
    list
        Sorted list of ``(year, month)`` tuples.

    Examples
    --------
    ```python
    src.available_months()[:2]   # [(2000, 1), (2000, 2)]
    ```
    """
    return sorted(self._index.keys())

close ¶

close() -> None

Release any open NetCDF file handles.

Safe to call more than once. Rasters are opened and closed per call to get_month, so this is a no-op in raster mode.

Source code in pycropwat/local.py

def close(self) -> None:
    """
    Release any open NetCDF file handles.

    Safe to call more than once. Rasters are opened and closed per call to
    ``get_month``, so this is a no-op in raster mode.
    """
    if self._dataset is not None:
        try:
            self._dataset.close()
        except Exception:  # pragma: no cover - already closed
            pass
        self._dataset = None
    self._data_array = None
    for dataset in self._file_cache.values():
        try:
            dataset.close()
        except Exception:  # pragma: no cover - already closed
            pass
    self._file_cache = {}

get_month ¶

get_month(year: int, month: int) -> Optional[xr.DataArray]

Read the precipitation grid for one month.

Parameters:

Name	Type	Description	Default
`year`	`int`	Calendar year.	required
`month`	`int`	Calendar month (1-12).	required

Returns:

Type	Description
`DataArray or None`	2-D array with dims `('y', 'x')`, dtype `float32`, values in mm (`scale_factor` applied), nodata as NaN, a descending `y` coordinate and the source CRS attached. Returns None when the month is missing.

Examples:

da = src.get_month(2005, 7)
if da is not None:
    print(da.dims, da.shape, float(da.mean()))

Source code in pycropwat/local.py

def get_month(self, year: int, month: int) -> Optional[xr.DataArray]:
    """
    Read the precipitation grid for one month.

    Parameters
    ----------

    year : int
        Calendar year.

    month : int
        Calendar month (1-12).

    Returns
    -------
    xr.DataArray or None
        2-D array with dims ``('y', 'x')``, dtype ``float32``, values in mm
        (``scale_factor`` applied), nodata as NaN, a descending ``y`` coordinate
        and the source CRS attached. Returns None when the month is missing.

    Examples
    --------
    ```python
    da = src.get_month(2005, 7)
    if da is not None:
        print(da.dims, da.shape, float(da.mean()))
    ```
    """
    year = int(year)
    month = int(month)
    if not 1 <= month <= 12:
        raise ValueError(f"month must be between 1 and 12, got {month}")

    if (year, month) not in self._index:
        logger.warning("No local precipitation data for %d-%02d", year, month)
        return None

    if self._kind == 'raster':
        return self._read_raster_month(year, month)
    return self._read_netcdf_month(year, month)

has_month ¶

has_month(year: int, month: int) -> bool

Check whether a given month is available.

Parameters:

Name	Type	Description	Default
`year`	`int`	Calendar year.	required
`month`	`int`	Calendar month (1-12).	required

Returns:

Type	Description
`bool`	True when the month can be read.

Source code in pycropwat/local.py

def has_month(self, year: int, month: int) -> bool:
    """
    Check whether a given month is available.

    Parameters
    ----------

    year : int
        Calendar year.

    month : int
        Calendar month (1-12).

    Returns
    -------
    bool
        True when the month can be read.
    """
    return (int(year), int(month)) in self._index

open_local_precipitation ¶

open_local_precipitation(path: Union[str, Path], **kwargs) -> LocalPrecipitationSource

Open a local precipitation dataset.

Thin convenience factory around LocalPrecipitationSource; every keyword argument is forwarded unchanged.

Parameters:

Name	Type	Description	Default
`path`	`str or Path`	A directory of monthly rasters, a single raster/NetCDF file, or a glob string.	required
`**kwargs`		Keyword arguments forwarded to `LocalPrecipitationSource` (`pattern`, `variable`, `scale_factor`, `nodata`, `crs`, `date_regex`, `time_dim`, `x_dim`, `y_dim`).	`{}`

Returns:

Type	Description
`LocalPrecipitationSource`	An indexed, ready-to-read precipitation source.

Examples:

from pycropwat.local import open_local_precipitation

src = open_local_precipitation('../pyCropWat_Data/Precip',
                               pattern='Precip_*.tif',
                               nodata=-9999)
da = src.get_month(2005, 7)
src.close()

Source code in pycropwat/local.py

def open_local_precipitation(path: Union[str, Path], **kwargs) -> LocalPrecipitationSource:
    """
    Open a local precipitation dataset.

    Thin convenience factory around ``LocalPrecipitationSource``; every keyword
    argument is forwarded unchanged.

    Parameters
    ----------

    path : str or Path
        A directory of monthly rasters, a single raster/NetCDF file, or a glob string.

    **kwargs
        Keyword arguments forwarded to ``LocalPrecipitationSource``
        (``pattern``, ``variable``, ``scale_factor``, ``nodata``, ``crs``,
        ``date_regex``, ``time_dim``, ``x_dim``, ``y_dim``).

    Returns
    -------
    LocalPrecipitationSource
        An indexed, ready-to-read precipitation source.

    Examples
    --------
    ```python
    from pycropwat.local import open_local_precipitation

    src = open_local_precipitation('../pyCropWat_Data/Precip',
                                   pattern='Precip_*.tif',
                                   nodata=-9999)
    da = src.get_month(2005, 7)
    src.close()
    ```
    """
    return LocalPrecipitationSource(path, **kwargs)

parse_year_month ¶

parse_year_month(name: Union[str, Path], date_regex: Optional[str] = None) -> Optional[Tuple[int, int]]

Extract a (year, month) pair from a file name or arbitrary string.

Layouts are tried most-specific-first and the last match in the string wins, so directory noise and prefixes such as Precip_ or effective_precip_ never confuse the parser. Recognised layouts:

Layout	Example
`YYYY_MM`	`Precip_2005_07.tif`
`YYYY-MM`	`precip-2005-07.nc`
`YYYYMM`	`pr200507.tif`
`YYYY.MM`	`pr.2005.07.tif`

Parameters:

Name	Type	Description	Default
`name`	`str or Path`	File name, file stem, full path or any other string to parse.	required
`date_regex`	`str`	Custom regular expression exposing named groups `year` and `month`. When given, the built-in layouts are not used.	`None`

Returns:

Type	Description
`tuple or None`	`(year, month)` with `1 <= month <= 12` and a plausible year (1700-2200), or `None` when nothing matched.

Raises:

Type	Description
`ValueError`	If `date_regex` is not a valid regular expression or lacks the `year` / `month` named groups.

Examples:

from pycropwat.local import parse_year_month

parse_year_month('Precip_2005_07.tif')          # (2005, 7)
parse_year_month('pr200507')                    # (2005, 7)
parse_year_month('/data/2019/x-2005-07.nc')     # (2005, 7)
parse_year_month('no_date_here.tif')            # None

# Custom layout: MM_YYYY
parse_year_month('rain_07_2005.tif',
                 date_regex=r'(?P<month>\d{2})_(?P<year>\d{4})')  # (2005, 7)

Source code in pycropwat/local.py

def parse_year_month(
    name: Union[str, Path],
    date_regex: Optional[str] = None
) -> Optional[Tuple[int, int]]:
    """
    Extract a ``(year, month)`` pair from a file name or arbitrary string.

    Layouts are tried most-specific-first and the **last** match in the string wins, so
    directory noise and prefixes such as ``Precip_`` or ``effective_precip_`` never
    confuse the parser. Recognised layouts:

    | Layout    | Example                    |
    |-----------|----------------------------|
    | ``YYYY_MM`` | ``Precip_2005_07.tif``   |
    | ``YYYY-MM`` | ``precip-2005-07.nc``    |
    | ``YYYYMM``  | ``pr200507.tif``         |
    | ``YYYY.MM`` | ``pr.2005.07.tif``       |

    Parameters
    ----------

    name : str or Path
        File name, file stem, full path or any other string to parse.

    date_regex : str, optional
        Custom regular expression exposing named groups ``year`` and ``month``.
        When given, the built-in layouts are not used.

    Returns
    -------
    tuple or None
        ``(year, month)`` with ``1 <= month <= 12`` and a plausible year
        (1700-2200), or ``None`` when nothing matched.

    Raises
    ------
    ValueError
        If ``date_regex`` is not a valid regular expression or lacks the
        ``year`` / ``month`` named groups.

    Examples
    --------
    ```python
    from pycropwat.local import parse_year_month

    parse_year_month('Precip_2005_07.tif')          # (2005, 7)
    parse_year_month('pr200507')                    # (2005, 7)
    parse_year_month('/data/2019/x-2005-07.nc')     # (2005, 7)
    parse_year_month('no_date_here.tif')            # None

    # Custom layout: MM_YYYY
    parse_year_month('rain_07_2005.tif',
                     date_regex=r'(?P<month>\\d{2})_(?P<year>\\d{4})')  # (2005, 7)
    ```
    """
    text = str(name)

    if date_regex is not None:
        try:
            pattern = re.compile(date_regex)
        except re.error as exc:
            raise ValueError(f"Invalid date_regex {date_regex!r}: {exc}") from exc
        group_names = pattern.groupindex
        if 'year' not in group_names or 'month' not in group_names:
            raise ValueError(
                f"date_regex must define named groups 'year' and 'month'. "
                f"Got groups: {sorted(group_names)}. "
                r"Example: r'(?P<year>\d{4})_(?P<month>\d{2})'"
            )
        patterns = (pattern,)
    else:
        patterns = _DATE_PATTERNS

    for pattern in patterns:
        matches = list(pattern.finditer(text))
        # Anchor on the LAST match, but fall back to earlier ones if it is implausible.
        for match in reversed(matches):
            try:
                year = int(match.group('year'))
                month = int(match.group('month'))
            except (TypeError, ValueError):
                continue
            if 1 <= month <= 12 and _MIN_YEAR <= year <= _MAX_YEAR:
                return year, month

    return None

Local Data Module¶

local ¶

LocalPrecipitationSource ¶

bounds property ¶

crs property ¶

files property ¶

kind property ¶

native_bounds property ¶

resolution property ¶

shape property ¶

year_range property ¶

__enter__ ¶

__exit__ ¶

__len__ ¶

__repr__ ¶

available_months ¶

close ¶

get_month ¶

has_month ¶

open_local_precipitation ¶

parse_year_month ¶

bounds `property` ¶

crs `property` ¶

files `property` ¶

kind `property` ¶

native_bounds `property` ¶

resolution `property` ¶

shape `property` ¶

year_range `property` ¶

enter ¶

exit ¶

len ¶

repr ¶