CSV format provider
The csv format interprets raw CSV bytes through Arrow and exposes a table to
SQL. It is a URL format provider, not a factory with a YAML configuration.
For pond files use csv:///path.csv; for host files use
host+csv:///path.csv, optionally rooted with -d DIRECTORY.
URL example
# Replace the example host path with a CSV file.
# All non-null defaults are explicit: comma (%2C), double quote (%22),
# header row, 8192-row batches, and 100 records for type inference.
# escape has no default character: omit it, rather than write escape=null.
pond cat 'host+csv:///tmp/readings.csv?delimiter=%2C"e=%22&has_header=true&batch_size=8192&schema_infer_max_records=100' --format=table
For an explicit backslash escape character, add &escape=%5C. That is an
example override, not the default. For pond files, remove the host+ layer.
Runnable example
Prerequisites: Bash and pond on PATH. No pond, credentials, or service is
required. SOURCE_DIR and WORK are initialized by the complete script below.
timestamp,temperature,humidity
2024-01-01T00:00:00Z,20.0,45
2024-01-01T01:00:00Z,25.0,44
2024-01-01T02:00:00Z,30.0,46
#!/bin/bash
set -euo pipefail
SOURCE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
if [[ -n "${EXAMPLE_OUTPUT:-}" ]]; then
mkdir "${EXAMPLE_OUTPUT}"
WORK="$(cd "${EXAMPLE_OUTPUT}" && pwd)"
else
WORK="$(mktemp -d "${TMPDIR:-/tmp}/watertown-csv.XXXXXX")"
fi
# docs:start query
pond cat -d "${SOURCE_DIR}" 'host+csv:///readings.csv' --format=table \
--sql 'SELECT timestamp, temperature, humidity FROM source WHERE temperature >= 25 ORDER BY timestamp' \
> "${WORK}/selected.txt"
pond cat -d "${SOURCE_DIR}" 'host+csv:///readings.csv' --format=table \
--sql 'SELECT COUNT(*) AS rows, SUM(temperature) AS total, AVG(humidity) AS mean_humidity FROM source' \
> "${WORK}/summary.txt"
pond cat -d "${SOURCE_DIR}" 'host+csv:///semicolon.csv?delimiter=;' --format=table \
--sql 'SELECT SUM(temperature) AS parameter_total FROM source' \
> "${WORK}/parameter.txt"
# docs:end query
cat "${WORK}/selected.txt" "${WORK}/summary.txt" "${WORK}/parameter.txt"
echo "Example output: ${WORK}"
Test 057 asserts the parsed timestamp/temperature/humidity columns, selected
rows (25,44) and (30,46), three input rows, temperature total 75, mean
humidity 45, and semicolon-file total 75.
Input, parameter input, complete script, download package.
URL parameters
Quote URLs containing query parameters so the shell does not interpret &.
For example, the tested ?delimiter=; selects a semicolon-separated input.
The generated shape below describes typed options before URL query decoding; URL values are strings parsed by the provider. Use ASCII delimiter/quote/escape characters.
Full configuration
Settings
| Field | Accepted type | Required | Default | Description |
|---|---|---|---|---|
batch_size | integer | no | 8192 | Number of records per batch (default: 8192) Constraints: format="uint"; minimum=0. |
delimiter | string | no | "," | Field delimiter (default: ',') Constraints: minLength=1; maxLength=1. |
escape | string | null | no | not specified | Escape character (default: None) Constraints: minLength=1; maxLength=1. |
has_header | boolean | no | true | Whether the file has a header row (default: true) |
quote | string | no | """ | Quote character (default: '"') Constraints: minLength=1; maxLength=1. |
schema_infer_max_records | integer | no | 100 | Number of rows to sample for schema inference (default: 100) Constraints: format="uint"; minimum=0. |
Use positive batch/inference sizes. Unknown options are ignored. Current limitation: an option-deserialization error falls back to the default options as a set, rather than reporting a configuration error. This pilot does not change runtime acceptance rules.
Types and errors
Arrow infers column types from sampled records; timestamp recognition depends on their contents. The fixture uses RFC 3339 UTC timestamps. A later value incompatible with the inferred schema can fail decoding; CSV does not impose a universal timestamp column or timezone.
Streaming decode uses a buffered initial chunk (up to 1 MiB) for inference, then feeds batches to Arrow. The separate schema-inference API reads the whole file into memory. Neither path should be described as unconditionally constant-memory inference.
Provider URL compression layers are +gzip, +zstd, and +bzip2, e.g.
host+csv+gzip:///readings.csv.gz. Compression and malformed-file cases are
not claimed as exercised by this pilot.
Related workflow
Continue with creating a timeseries and compare synthetic output.