CSV format provider

The csv format interprets raw CSV bytes through Arrow and exposes a table to SQL. It is a URL format provider, not a factory with a YAML configuration. For pond files use csv:///path.csv; for host files use host+csv:///path.csv, optionally rooted with -d DIRECTORY.

URL example

# Replace the example host path with a CSV file.
# All non-null defaults are explicit: comma (%2C), double quote (%22),
# header row, 8192-row batches, and 100 records for type inference.
# escape has no default character: omit it, rather than write escape=null.
pond cat 'host+csv:///tmp/readings.csv?delimiter=%2C&quote=%22&has_header=true&batch_size=8192&schema_infer_max_records=100' --format=table

For an explicit backslash escape character, add &escape=%5C. That is an example override, not the default. For pond files, remove the host+ layer.

Runnable example

Prerequisites: Bash and pond on PATH. No pond, credentials, or service is required. SOURCE_DIR and WORK are initialized by the complete script below.

timestamp,temperature,humidity
2024-01-01T00:00:00Z,20.0,45
2024-01-01T01:00:00Z,25.0,44
2024-01-01T02:00:00Z,30.0,46
#!/bin/bash
set -euo pipefail
SOURCE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
if [[ -n "${EXAMPLE_OUTPUT:-}" ]]; then
    mkdir "${EXAMPLE_OUTPUT}"
    WORK="$(cd "${EXAMPLE_OUTPUT}" && pwd)"
else
    WORK="$(mktemp -d "${TMPDIR:-/tmp}/watertown-csv.XXXXXX")"
fi

# docs:start query
pond cat -d "${SOURCE_DIR}" 'host+csv:///readings.csv' --format=table \
    --sql 'SELECT timestamp, temperature, humidity FROM source WHERE temperature >= 25 ORDER BY timestamp' \
    > "${WORK}/selected.txt"
pond cat -d "${SOURCE_DIR}" 'host+csv:///readings.csv' --format=table \
    --sql 'SELECT COUNT(*) AS rows, SUM(temperature) AS total, AVG(humidity) AS mean_humidity FROM source' \
    > "${WORK}/summary.txt"
pond cat -d "${SOURCE_DIR}" 'host+csv:///semicolon.csv?delimiter=;' --format=table \
    --sql 'SELECT SUM(temperature) AS parameter_total FROM source' \
    > "${WORK}/parameter.txt"
# docs:end query
cat "${WORK}/selected.txt" "${WORK}/summary.txt" "${WORK}/parameter.txt"
echo "Example output: ${WORK}"

Test 057 asserts the parsed timestamp/temperature/humidity columns, selected rows (25,44) and (30,46), three input rows, temperature total 75, mean humidity 45, and semicolon-file total 75.

Input, parameter input, complete script, download package.

URL parameters

Quote URLs containing query parameters so the shell does not interpret &. For example, the tested ?delimiter=; selects a semicolon-separated input.

The generated shape below describes typed options before URL query decoding; URL values are strings parsed by the provider. Use ASCII delimiter/quote/escape characters.

Full configuration

Settings

FieldAccepted typeRequiredDefaultDescription
batch_sizeintegerno8192Number of records per batch (default: 8192) Constraints: format="uint"; minimum=0.
delimiterstringno","Field delimiter (default: ',') Constraints: minLength=1; maxLength=1.
escapestring | nullnonot specifiedEscape character (default: None) Constraints: minLength=1; maxLength=1.
has_headerbooleannotrueWhether the file has a header row (default: true)
quotestringno"""Quote character (default: '"') Constraints: minLength=1; maxLength=1.
schema_infer_max_recordsintegerno100Number of rows to sample for schema inference (default: 100) Constraints: format="uint"; minimum=0.

Use positive batch/inference sizes. Unknown options are ignored. Current limitation: an option-deserialization error falls back to the default options as a set, rather than reporting a configuration error. This pilot does not change runtime acceptance rules.

Types and errors

Arrow infers column types from sampled records; timestamp recognition depends on their contents. The fixture uses RFC 3339 UTC timestamps. A later value incompatible with the inferred schema can fail decoding; CSV does not impose a universal timestamp column or timezone.

Streaming decode uses a buffered initial chunk (up to 1 MiB) for inference, then feeds batches to Arrow. The separate schema-inference API reads the whole file into memory. Neither path should be described as unconditionally constant-memory inference.

Provider URL compression layers are +gzip, +zstd, and +bzip2, e.g. host+csv+gzip:///readings.csv.gz. Compression and malformed-file cases are not claimed as exercised by this pilot.

Continue with creating a timeseries and compare synthetic output.