aft: ASCII Framed Tables

Tabular text files (like CSV or TSV), but using special ASCII characters to distinguish header, rows, and fields. For storing tabular data in files, or for simple structured data transfer between programs and over networks.

file suffix: .aft

mimetype (speculative): text/x-aft

Format

Format is:

• start header: 0x01 byte (ANSI "Start of Header")
• header rows, with same width/structure as table (see below)
  • first row, required, is column names. if blank name, column is unnamed. duplicate names are ambiguous: pass-through by some tools, forbidden by check
  • second row indicates column types (see below)
  • third row indicuates current sort order
• start body: 0x02 byte (ANSI "Start of Text")
• rows are lines ending with 0x1D (ANSI "Group Separator"), immediately followed by a newline. TBD how strict about including the newline. Other newlines in the text do not incidate a new row, they are part of the cell ("record" or "unit") value
• within a row, columns are separated by 0x1E ("Record Separator"). Within columns, records can optionally have additional structure, separated by 0x1F ("Unit Separator")

The special characters are:

• 0x01 1 \001 SOH ^A: "Start of Header"
• 0x02 2 \002 STX ^B: "Start of Text"
• 0x03 3 \003 ETX ^C: "End of Text"
• 0x1C 28 \034 FS ^\: "File Separator"
• 0x1D 29 \035 GS ^]: "Group Separator"
• 0x1E 30 \036 RS ^^: "Record Separator"
• 0x1F 31 \037 US ^_: "Unit Separator"

Tools and file readers check the start of stream. If it is 0x01, can treat as aft; otherwise treat as regular text. 0x02 terminates header section and starts contents. Header records could somehow include:

• column names
• column types (default is string)
• indicate sort order (columns, asc/desc)

General structure of tables is:

• units: for arrays of values in a cell (no other structure) => could be "array" (variable-length, single type) or "tuple" (fixed-length, names and types for each slot)
• records: for cells in 2-dimensional grid
• groups: rows of table (or, use newline, so regular tools work?)
• file: unused; could be "sheets"? sort of redundant with "end of text" character

Comments and annotations are not supported.

Default strings must not contain any of the special characters. This must be enforced by code doing the encoding, and requires a full pass of every value. In this mode, strings which include the special characters may not be included and must raise an error. For UTF-8 strings which may include the characters, an escaped column type is allowed, which allows arbitrary JSON-style character escapes (like '\u01`). The downside with this mode is that conversion and processing may be slower.

Extras

Text editor support: coloring, easy entry of special chars, highlighting errors, tab-separation.

Unfortunately, my terminal (xterm) interprets Ctrl-V Ctrl-_ as "resize smaller".

Column Types

Encoding is that of JSON or TOML. Values can be:

• string (UTF-8) (default if not specified)
• escaped byte-string, JSON-style
• base64 encoded bytes
• integer number
• floating point number
• boolean
• date, timestamp (ISO format)
• null

Usage Examples

cargo run --bin aft-demo | cat -v
cargo run --bin aft-demo | cut -f1 -d$'\036' | cat -v

Use Cases

• (structured) logging
• (structured) command output
• data munging pipelines and intermediate formats
• datastore imports and exports
• REST API output (via content negotiation)
• efficient
• sorted static lookup tables

Ideas

• sub-types for columns. eg, "uint+bytes"
• sort order

Similar To

• Hadoop formats?
• "column families" in datastores
• CDX (web archiving), particularly CDXJ (JSON-ish)

Open Questions

• should windows-style terminators be allowed? or strictly single-newline-character after the Group Separator?
• should SOH and STX characters be followed by newline? optionally? this may make some command-line stuff easier, but adds parsing complexity (how to represent leading whitespace in first column?)
• what about newlines in values? should there be a mode where we use record or group separator instead of newline? single mode is much better, but compatibility with existing UNIX-y newline-oriented stuff also compelling. Hrm.
• should first line in file starting with # be ignored? for future extensibility
• should the default mode be escaped strings, or strings with no special characters allowed? may not matter for the common case of no special characters.
• should additional constraints be placed on header encoding? eg, that they must be escaped or not have any ASCII special characters. this might make parsing to determine validity and mode easier
• should there be fast ASCII-only, no special characters mode? would that provide even more speed? daft?