Archive¶

dpc_radar.archive ¶

A local archive of the «Merging», and hourly rain corrected with the gauges.

The DPC API keeps about five months of frames, then they are gone. The radar-only history (IT-DPC-SRI, :mod:dpc_radar.history) under-estimates: in Apulia in September 2016 it read between 10 % and 80 % of the rain gauges. The «Merging» is the correction with the gauges, and nobody archives it publicly: whatever is not saved in time is lost, one month at a time.

Two products are kept, because the shortest «Merging» is three hours long:

CUM3 at 00, 03, ..., 21 UTC: three-hour blocks that do not overlap, radar and gauges, 0.01° in EPSG:4326;
SRT1 at every full hour: the radar-only rain of that hour, 1 km.

The rain of one hour is SRT1 rescaled so that the three hours of a block add up exactly to the CUM3: the «Merging» says how much, the radar says when. If the radar saw nothing in a block (or missed an hour of it) and the gauges did, the block is split in equal parts; without CUM3, SRT1 stays as it is; without either, the hour is not measured (None), which is not the same as dry.

Layout: <folder>/<PRODUCT>/<YYYY>/<MM>/<DD>/<PRODUCT>_<YYYYMMDDhhmm>.tif, plus <folder>/<PRODUCT>/absent.txt with the frames the DPC no longer keeps. It is the layout of Limen's archive, so an existing one can be read as is (its assenti.txt is read too).

Whatever goes wrong with one frame stays with that frame: it is counted, or read as not there, and the reason goes to the dpc_radar.archive logger as a warning. Attach a handler to see which frames and why.

ArchiveReport `dataclass` ¶

What one :func:archive run did.

MergingArchive ¶

Hourly rain corrected with the gauges, read from an archive on disk.

Frames are kept in memory, cache_frames at most (an SRT1 frame is about 6.7 MB as float32): the same frames are read for every region of a run. One instance can be shared between threads: the cache is locked, and frames are decoded outside the lock.

read ¶

read(product: str, time: datetime) -> RainGrid | None

One archived frame, or None if it is not there or not readable.

hourly ¶

hourly(points: list[tuple[float, float]], start: datetime, end: datetime) -> dict[datetime, list[float | None]]

Rain of every hour (h-1, h] in (start, end], per lon/lat point (mm).

path ¶

path(folder: Path, product: str, time: datetime) -> Path

Where a frame lives in the archive.

times ¶

times(product: str, start: datetime, end: datetime) -> list[datetime]

The archived times of product in [start, end], oldest first.

archive `async` ¶

archive(folder: Path, radar: RadarClient, *, start: datetime, end: datetime, limit: int, newest_first: bool = False, now: datetime | None = None) -> ArchiveReport

Download the CUM3 and SRT1 frames of [start, end] that the archive lacks.

Oldest first, because the oldest is the one the DPC will lose first; newest_first for a run that must keep up with the last hours before the limit is spent on old frames. Idempotent: a frame on disk is not downloaded again, and one the DPC no longer keeps is written to absent.txt and not asked again. A frame younger than :data:ABSENT_AFTER that the API calls gone is not written down: it may just not be published yet. Stops after limit requests. A frame that fails for any other reason is counted, logged and retried on the next run.

With a send hook on the client, a failure must surface as an :class:httpx.HTTPError, a :class:DpcRadarError or an :class:OSError (TimeoutError is one) to count as one failed frame. Anything else is taken for a bug and ends the run.

split_block ¶

split_block(merging: float | None, radar: list[float | None], j: int) -> float | None

The share of hour j of a block in its «Merging» total.

radar holds the SRT1 rain of each hour of the block. The «Merging» says how much, the radar says when; if the radar saw nothing, or missed an hour, the total is split in equal parts.