AnyLog only. Aggregation functions are not available in EdgeLake.

Aggregation functions summarize streaming data over defined time intervals. For each table, users configure the interval duration and the number of intervals to retain, enabling continuous, rolling aggregations.

As data is ingested, aggregation functions continuously compute statistics such as count, sum, average, minimum, and maximum for each interval. The resulting aggregated values can be queried directly or used to influence database updates, alerts, and monitoring logic.

This approach enables efficient real-time analytics on streaming data without requiring downstream batch processing or data centralization.

Aggregation time intervals

Aggregation intervals define how streaming data is grouped and processed before database ingestion. Each interval represents a time window during which incoming data is collected and evaluated by the aggregation functions (e.g., min, max, avg, count).

Key Concepts

Processing-Time Based
Aggregation intervals are determined by the system’s processing timeline (when data is received), not by the timestamp within the source data.
Data Grouping All streaming data received during a given interval is grouped together and processed as a single unit, regardless of the original timestamp embedded in the data.
Independence from Source Time Although it is generally assumed that source data timestamps increase over time, this is not guaranteed.
In practice:
- Data may arrive out of order
- Multiple records with similar or identical timestamps may be processed in different aggregation intervals
- Late-arriving data may be included in a later interval, even if its timestamp is earlier

Implications

Aggregation results reflect when data was processed, not strictly when it was generated.
The same logical time range (based on source timestamps) may be split across multiple aggregation intervals.
Accurate time-based analysis using source timestamps may require additional handling.

Viewing Aggregation Results

Two commands provide different perspectives on the aggregated data:

get aggregation - Displays results organized by aggregation (processing) time intervals.
get aggregation by time - Displays results ordered by the source data timestamps, allowing analysis based on the original event time rather than processing time.

Aggregations and DBMS Operations

Aggregations can be configured to support different data-processing modes:

Source data ingestion enabled (default) Raw (source) data continues to be written to the database while aggregations are computed in parallel.
Source data ingestion disabled Raw (source) data is not written to the database while aggregations are computed in parallel.
In memory Aggregation (default)
Aggregations maintained in memory only
Aggregation ingestion enabled (default)
Aggregations written to the database

These modes allow users to balance storage cost, write volume, and query granularity while preserving real-time insight.

Usage:

set aggregation ingest where dbms = [dbms name] and table = [table name] and source = [true/false] and derived = [true/false]

Notes:

source controls ingestion of raw (source) data.
derived controls ingestion of aggregation results.
Defaults:
- source = true
- derived = false
When source ingestion is disabled, only aggregation results can be persisted.
This enables aggregation-driven storage, reduced write load, and real-time summarized data.

Partitioning and Aggregations

If aggregation output is stored in the same physical database as the source time-series data (see Aggregations and DBMS Operations above), avoid partitioning that database with a single wildcard rule (partition [dbms] * using [column] by [interval] — see Table Partitioning A database-wide rule applies the same interval to every table in that database. I doesn’t distinguish between raw source tables and aggregation output tables : the two typically need very different retention.

Instead, disable the database-wide rule and partition each table individually, so raw data and its aggregation output can each use an interval and retention window suited to that table:

curl -X POST http://[ip]:[operator port] \
   -H "command: partition my_data ping_sensor using timestamp by 1 day" \
   -H "AnyLog-Agent: AnyLog/1.23"

Don’t apply the same cleanup schedule to the aggregation (target) table as you do to the raw source table. The raw table’s partition-drop schedule is usually tuned to expire data quickly; the aggregation table exists specifically to preserve a compact historical summary after the raw data has aged out. Reusing the raw table’s short keep value on the aggregation table’s own partitions will cause that summarized history to disappear on the same short timeline — defeating the point of aggregating in the first place. Aggregation tables can still be partitioned; just give them a keep value (or interval) that reflects how long you actually want the summarized history retained, independent of the source table’s schedule.

When defining partitioning manually per table, also define the cleanup schedule per table (rather than relying on one general schedule for the whole database):

curl -X POST http://[ip]:[operator port] \
   -H 'command: schedule time=30 days and name="Drop Partitions - [db name].[table]" task drop partition where dbms=[db name] and table=[table name] and keep=[number of partitions]' \
   -H "AnyLog-Agent: AnyLog/1.23"

The advantage of partitioning per table is granularity: a database-wide (*) rule forces every table in that database onto the same interval unless you additionally define a table-specific partition — at which point the table-specific definition takes precedence for that table.

View Aggregation Ingest Declaration

Usage:

get aggregation ingest

The command above returns the mode of database operation for source and derived data.

How They Work:

Users define a time interval (e.g., 1 minute, 5 minutes, or hourly).
Users define the number of intervals to maintain.
The system continuously aggregates incoming data within each interval.
Aggregated values can be used in real-time for:
- Querying the current trends
- Triggering alerts or monitoring changes
- Impacting database updates

For high-volume data streams, aggregation encoding can replace raw data with compressed representations, significantly reducing data volume.

Declaring Aggregations

Using the command set aggregation users can track data streamed to a node.

Monitoring applies per DBMS and table, aggregating values from a specified column across predefined time intervals.

Aggregated Metrics

Usage:

set aggregation where dbms = [dbms name]
      and table = [table name]
      and intervals = [count]
      and time = [interval time]
      and time_column = [time column name]
      and value_column = [value column name]
      and target_dbms = [target dbms name]
      and target_table = [target table name]

Command option	Default	Details
dbms		The name of the database that hosts the table’s data.
table		The data table name. If table name is not provided, all the tables associated to the database are monitored using the database definitions.
intervals	10	The number of intervals to keep.
time	1 minute	The length of the interval expressed in one of the following: seconds, minutes, hours, days.
time_column	“timestamp”	The name of the time column.
value_column	“value”	The name of the column being monitored.
target_dbms	[dbms]	The name of the dbms to host the aggregation data. Defaults to the same as the source dbms if not provided.
target_table	[table]	The name of the table to host the aggregation data. Defaults to the same as the source table.

Example:

set aggregation where dbms = dmci and table = sensor_table and intervals = 10 and time = 1 minute and time_column = timestamp and value_column = value

Declaring Thresholds

Users can declare thresholds on each stream. These thresholds can be referenced by the rule engine to impact the processing of the stream and trigger operations that consider the thresholds.

Usage:

set aggregation thresholds where dbms = [dbms name] and table = [table name] and column = [column name] and min = [min value] and max = [max value] and avg = [average value] and count = [events count]

Reset aggregations

The command reset aggregations deletes the aggregation declarations.
Usage:

reset aggregations where dbms = [dbms name] and table = [table name] and value_column = [column name]

Aggregation encoding

The command set aggregations encoding applies encoding on the values assigned to each time interval.
The encoding is represented by a new data set associated to a new schema. The table name with the encoding format is using the original table name, prefixed by the encoding type. For example, if the source data is assigned to a table named my_table and the encoding type is arle, the table with the encoded data is called arle_my_table.

Usage:

set aggregations encodeing where dbms = lsl_demo and table = ping_sensor and encoding = [encoding type] and tolerance = [value]

Encoding types:

None - No encoding (default).
bounds - all entries in the time interval are replaced with a single entry representing:
- timestamp - The earliest date and time of the entries represented in the interval.
- end_interval - The latest date and time of the entries represented in the interval.
- min_val - The lowest value recorded within the time interval.
- max_val - The highest value recorded within the time interval.
- avg_val - The average of the values recorded within the time interval.
- events - The number of events recorded within the time interval.
arle - Approximated Run-Length Encoding, the entries in the time interval are represented in a sequence of entries. Each entry includes:
- timestamp - The earliest date and time of the entries represented in the interval.
- end_interval - The latest date and time of the entries represented in the interval.
- avg_val - average of grouped values.
- events - The number of events recorded within the time interval.

tolerance
With Approximated Run-Length Encoding, allowable difference between consecutive values while treating them as part of the same group or segment.
Tolerance is represented as percentage difference.

Example:

set aggregation encoding where dbms = lsl_demo_ok and table = rand_table and encoding = arle and tolerance = 5

Retrieve Aggregations

The command get aggregation retrieves the aggregation definitions and runtime statistics configured with the set aggregation command.

Syntax

get aggregation [where <condition>]

Keyword	Description
dbms	The name of the source DBMS processed in the aggregation.
table	The name of the source table processed in the aggregation.
value_column	The name of the value column processed in the aggregation.
format	Output format. Supported values: table or json. Default is table.
function	Aggregation function requested. Supported values: min, max, avg, count.
round	If set to true, rounds min, max, and avg values to 3 decimal digits.

Usage Examples

get aggregation 
get aggregation where dbms = orics and round = true
get aggregation where dbms = orics and table = r_50
get aggregation where dbms = orics and table = r_50 and value_column = seal_storage

Sample output

DBMS  Target DBMS Table Target Table          Value Column      interval H:M:S       Events/sec Count Min     Max     Avg
-----|-----------|-----|---------------------|----------------|--------|-----------|----------|-----|-------|-------|-------|
orics|orics_agg  |r_50 |r_50_seal_storage    |seal_storage    |       0|0:3:0      |      0.17|   10| 11.020| 93.030| 39.064|
orics|orics_agg  |r_50 |r_50_heater1_temp    |heater1_temp    |       0|0:0:0      |      0.17|   10|103.010|299.290|202.909|