Time Series Optimizations

Time-series optimizations​

QuestDB is specifically designed for time-series, and it provides several optimizations, like a designated timestamp, sequential reads, materialized-views, or in-memory processing.

Designated timestamp​

  • Timestamp sorting: Data stores in order of incremental timestamp. Since ingestion is usually chronological, the system uses a fast append-only strategy, except for updates and out-of-order data.

  • Rapid interval queries and sequential reads: Sorted data lets the system quickly locate the start and end of data files, which speeds up interval queries. When data is accessed by increasing timestamp, reads are sequential for each column file, which makes I/O very efficient.

Interval scan
Interval scan
  • Out-of-order data: When data arrives out of order, QuestDB rearranges it to maintain timestamp order. The engine splits partitions to minimize write amplification and compacts them in the background.

Data partitioning and sequential reads​

Diagram of data column files and how they are partitioned to form a table
Diagram of data column files and how they are partitioned to form a table

  • Partition pruning: The design lets the engine skip partitions that fall outside query filters. Combined with incremental timestamp sorting, this reduces latency.

  • Lifecycle policies: The system can delete partitions manually or automatically via TTL. It also supports detaching or attaching partitions using SQL commands.

Materialized views​

  • Materialize views are auto-refreshing tables storing the pre-computed results of a query. Unlike regular views, which compute their results at query time, materialized views persist their data to disk, making them particularly efficient for expensive aggregate queries that are run frequently.

  • QuestDB supports materialized views for SAMPLE BY queries, including those joining with other tables.

  • Materialized sampled intervals are automatically refreshed whenever the base table receivews new or updated rows.

  • Materialized views can be chained, with the output of one being the input of another one, and support TTLs for lifecycle management.

In-memory processing​

  • Caching: The engine uses the OS cache to access recent and frequently accessed data in memory, reducing disk reads.

  • Off-heap buffers: Off-heap memory, managed via memory mapping and direct allocation, avoids garbage collection overhead.

  • Optimized in-memory handling: Apart from using CPU-level optimizations such as SIMD, QuestDB uses specialized hash tables (all of them with open addressing and linear probing), and implements algorithms for reducing the memory footprint of many operations.

  • Custom memory layout for different data types: Specialized data types, like Symbol, VARCHAR, or UUID, are designed to use minimal disk and memory. For example, char sequences shorter than 9 bytes are fully inlined within our VARCHAR header and do not occupy any additional data space.

Internal Representation of the VARCHAR data type

Varchar header (column file):
+------------+-------------------+-------------------+
| 32 bits    | 48 bits           | 48 bits           |
| len + flags| prefix            | offset            |
+------------+-------------------+-------------------+
                                     │
+------------------------------------+ points to
│
▼
Varchar data (column file):
+---+---+---+---+---+---+---+---+---+---+---+
| H | e | l | l | o |   | w | o | r | l | d |
+---+---+---+---+---+---+---+---+---+---+---+

Next Steps​