Temporal reduction directory

Overview

temporal-reduce creates a dynamic directory containing one reduced series per configured resolution and output pattern.

Example configuration

in_pattern: series:///sensors/*  # Required: example source pattern.
out_pattern: "$0"               # Required: example output name from first capture.
time_column: timestamp          # Required: source event-time column.
group_by: []                    # Default: no additional grouping columns.
resolutions: [1h, 6h]            # Required: example aggregation windows.
aggregations:                   # Required: one or more aggregation operations.
  - type: avg                   # Required: example operation.
    columns: null               # Default: all eligible numeric columns.
transforms: null                # Default: no input transforms.
output_aliases: null            # Default: names such as temperature.avg.
allowed_lateness: 1d            # Effective default: one-day hot window.
seal_target_bytes: 1048576       # Effective default: 1 MiB before sealing.
max_live_segments: 50           # Effective default: compaction backstop.

Use columns: [temperature] to restrict the aggregation and, for example, output_aliases: {temperature.avg: temperature} to rename its output. Omitted or null cache settings resolve to the explicit defaults shown above.

Use and behavior

Install with pond mknod temporal-reduce /derived/reduced --config-path reduce.yaml, then list and query its children. Buckets are aligned time windows; a sample on the next boundary belongs to that next bucket. Reduction is aggregation, not interpolation.

Incremental caches use source versions and event-time metadata. Missing bounds can force full-history invalidation. Late backfill unseals and recomputes affected segments rather than silently dropping data.

Sealing policy

Older eligible buckets move from the recomputed hot file into reusable segment files when the byte target is reached or one day of sealable backlog has accumulated. The byte target defaults to 1 MiB. The backlog is measured up to the lateness-adjusted watermark, so it is additional to the recent allowed_lateness window, not a replacement for it.

With the default one-day lateness, regularly updated sparse series generally keep roughly one to two days hot, subject to bucket granularity and build cadence. This limits excess event-time duration, not absolute input work. Segment compaction still controls file count.

Changing this policy does not invalidate existing caches. A changed build may need to process an older oversized hot window once before advancing its sealed frontier. No source history is deleted, and late corrections remain supported.

For format-backed inputs such as csv:// and jsonlogs://, replacement files (FilePhysicalVersion) contribute only their current content. Replacing a file invalidates its previous aggregates; stored revisions are not extra observations. Append-only file series continue to contribute all live chunks.

Grouping dimensions

Set group_by: [site_id, page_uri] to produce one row per time bucket and distinct combination of those source columns. Keys retain their names and types, including null values; null keys group together as in SQL. With an empty or omitted group_by, existing time-only reduction is unchanged.

Grouping columns are exact names, not patterns or SQL expressions. They are carried through shared cached partials, coarser resolutions, and late-data recomputation. Changing the grouping selects a separate cache identity. Default aggregation-column discovery excludes the grouping keys.

The input table schema must contain the keys. Missing keys are errors, not an implicit null group; an unmatched source cannot supply their types. Keys must be unique and cannot conflict with the timestamp or aggregate outputs. time_bucket and names beginning __p_ are reserved internal names.

For daily event counts, combine group_by with resolutions: [1d], aggregations: [{type: count, columns: ["*"]}], and output_aliases: {timestamp.count: views}. Group cardinality determines the number of aggregate rows; grouping does not impose a cardinality limit.

Full configuration

Settings

FieldAccepted typeRequiredDefaultDescription
aggregationsarray of Aggregationyesnot specifiedAggregation operations to perform
allowed_latenessstring | nullnonot specifiedMaximum allowed lateness for the segment partial-aggregate cache: how far behind the newest bucket a late/backfilled sample may still land before its output bucket is treated as sealed. Buckets older than newest_bucket - allowed_lateness are frozen into immutable segment files and never recomputed, which bounds the rollup's working set to this window instead of all history. Parsed with humantime::parse_duration; defaults to one day. Data arriving older than the sealed watermark is not an error: the segments covering it are unsealed and recomputed from source. Changing this value invalidates the merged (segment) cache, which is rebuilt from the cached source versions.
group_byarray of stringnonot specifiedExact source columns retained as grouping keys alongside the time bucket.
in_patternWatertown Urlyesnot specifiedInput URL pattern to match source files (supports glob patterns)
max_live_segmentsinteger | nullnonot specifiedMaximum number of segment files to leave live per resolution before compaction merges them regardless of size class. Defaults to [crate::partial_aggregate_cache::MAX_LIVE_SEGMENTS]. Like seal_target_bytes, changing it invalidates nothing. Constraints: format="uint"; minimum=0.
out_patternstringyesnot specifiedOutput basename pattern using captured groups (e.g., "$0", "$1") - no URL scheme
output_aliasesobject | nullnonot specifiedOptional final output aliases keyed by the default <column>.<aggregate> name.
resolutionsarray of stringyesnot specifiedList of temporal resolutions (e.g., "1h", "6h", "1d") Parsed using humantime::parse_duration
seal_target_bytesinteger | nullnonot specifiedMinimum accumulated size before frozen buckets are sealed into an immutable segment file. Accepts a byte count; defaults to [crate::partial_aggregate_cache::SEAL_TARGET_BYTES]. Unlike allowed_lateness, changing this does NOT invalidate the cache: a segment already sealed is correct at any target, so the new value simply governs the next seal. Exposed mainly so the tradeoff can be measured against real data -- larger means fewer files but a heavier hot-window recompute each build. The fixed sealable-backlog span backstop still applies when output remains below this byte target. Constraints: format="uint64"; minimum=0.
time_columnstringyesnot specifiedName of the timestamp column
transformsarray | nullnonot specifiedOptional list of table transform factory paths to apply to input TableProvider. Transforms are applied in order before SQL execution. Each transform is a path like "/etc/hydro_rename"

Aggregation

Configuration for a single aggregation operation

FieldAccepted typeRequiredDefaultDescription
columnsarray | nullnonullColumns to apply the aggregation to If not specified, applies to all numeric columns Use ["*"] for count operations
typeAggregation Typeyesnot specifiedType of aggregation (avg, min, max, count, sum)

Aggregation Type

Aggregation types supported by the temporal reduce factory

Accepted value: "avg" | "min" | "max" | "count" | "sum".

Watertown Url

Accepted value: string.