Map your custom data source

Eyer will work on any time-series data (metric + timestamp). You can connect open source agents like Influx Telegraf or Prometheus to support your technologies and applications, and you can also create custom agents / scripts that feed time-series data to Eyer.

To connect your custom data, you need to create a mapping from your source data to the Eyer internal format.

Requirements

Your data (from open source agent / custom agent / script) output must be in JSON format
Eyer currently only support metrics, not logs. If you work on logs, you can convert information of interest to metrics (e.g. number of errors matching a certain regex condition).
The metrics must have a timestamp in UNIX format (typically the time of metric collection)
The data must be mapped to the Eyer internal data structure (Node/metrics)

Input data example (coming from Influx Telegraf)

The example below uses Influx Telegraf to collect some database metrics, and send them over HTTPS to the Eyer API. What you see below is the JSON sent from Influx Telegraf. The “fields” section can repeat, so a request can contain metrics from multiple database servers.

{
    "metrics":
    [
        {
            "fields":
            {
                "disk_reads_mean": 1.0121461361787716,
                "disk_writes_per_sec_mean": 0.8571885607181682,
                "last_Run_mean": 1740047577489,
                "log_bytes_flushed_mean": 3.9596305549082618,
                "logical_reads_mean": 914.9966524003087,
                "total_sessions_mean": 14,
                "waits_total": 0,
                "waits_total_mean": 7.3
            },
            "name": "dbwatch",
            "tags":
            {
                "host": "testmachine1",
                "instance": "Database1",
                "url": "http://localhost:8080/dashboard/dbWatch%2001/data/HkRmUaLz0Z.telegraf.json"
            },
            "timestamp": 1740047760
        }
    ]
}

Mapping example with explanation

We take the example above, and map it to the Eyer internal format. Below you see the mapping (in JSON) , and then the mapping explained with functions in a table.

{
    "selector":
    [
        "tags.host",
        "tags.instance",
        "timestamp"
    ],
    "type": "telegraf.waitstats",
    "id_selector": "tags.instance",
    "root": "metrics",
    "system": "tags.host",
    "mappings":
    [
        {
            "type": "telegraf.waitstats.waits",
            "guid": "16e41dba-e6a4-4427-b753-cf6eee41e384",
            "name": "dbWatch waits",
            "timestamp": "timestamp",
            "fields":
            [
                {
                    "name": "fields.disk_reads_mean",
                    "display": "Disk reads",
                    "type": "float",
                    "aggregation": "avg",
                    "guid": "9ed41a78-3094-4e41-98b5-b1f98f63dc3a"
                },
                {
                    "name": "fields.disk_writes_per_sec_mean",
                    "display": "Disk writes per second",
                    "type": "float",
                    "aggregation": "avg",
                    "guid": "1203a428-0b80-4240-8c1b-f7fa2665ec22"
                },
                {
                    "name": "fields.log_bytes_flushed_mean",
                    "display": "Log bytes flushed",
                    "type": "float",
                    "aggregation": "avg",
                    "guid": "2add293a-446a-468e-91c3-8f164fe697c8"
                },
                {
                    "name": "fields.logical_reads_mean",
                    "display": "Logical reads",
                    "type": "float",
                    "aggregation": "avg",
                    "guid": "7acb184d-c4ce-4287-ab4a-f63f15976bac"
                },
                {
                    "name": "fields.total_sessions_mean",
                    "display": "Total sessions",
                    "type": "int",
                    "aggregation": "avg",
                    "guid": "30c71236-48ad-4fa8-b282-f587ffe8b4b9"
                },
                {
                    "name": "fields.waits_total_mean",
                    "display": "Waits total",
                    "type": "float",
                    "aggregation": "avg",
                    "guid": "1f2a5a97-338b-4a92-8846-f27cd9f6b71f"
                }
            ],
            "name_map": "tags.instance"
        }
    ],
    "skip_merge": true
}

Mapping	Comments

Mapping	Comments
selector:	A selectors array used for regrouping the data (e.g. if a single element of the array in the input contains multiple data points for different timestamps).
[
tags.host,	Input field to use for grouping.
. . .
],
type: "telegraf.waitstats",	The internal `type` that is used to distinguish between parsing rules. The suffix of the `type` is used for registering an API endpoing (e.g. `/api/telegraf/waitstats`). Should be `alpha`.
id_selector: "tags.instance",	Additional selector used for uniqueness checks of `nodes`. Used together with the `node` name (see `name_map` field) to guarantee that the node is unique within Eyer.
root: "metrics",	The root object to start the parsing.
system: "tags.host",	The object to pick up the value for `system` from.
mappings:	Mappings define `nodes` and `stat points` that should be created based on the input. Each element corresponds to a single `node type` and can have multiple `stat types`. `node types` have `1-to-many` relation with `nodes`, so multiple `nodes` of the same `node type` can be created.
[
{
type: "telegraf.waitstats.waits",	Internal identification for `node type`. Should be unique for mapping. Should be prefixed with the value of the `type` field.
guid: "16e41dba-e6a4-4427-b753-cf6eee41e384",	(Optional) Internal UUID, which is unique per `node type`. If missing, a UUID is created based on `node type` value.
name: "dbWatch waits",	User-friendly name per `node type`.
timestamp: "timestamp",	The field to pick the `timestamp` from. For telegraf type of mappings we accept timestamp in seconds.
fields:	An array of fields, each one constituting a single `stat type`. New `stat types` can be added to `node type`, but existing `stat types` cannot be altered or deleted if the `node` of corresponding `node type` was already created.
[
{
name: "fields.disk_reads_mean",	Internal name unique for each stat type. Should correspond to a field in the input that the value for the metric will be picked up from.
display: "Disk reads",	User friendly name used for display.
type: "float",	User friendly string representing a data type (e.g. “Count”, etc.) used for display.
aggregation: "avg",	Aggregation. `avg`, `max` and `sum` are supported, where `avg` will be aggregated on average, `max` - maximum value and `sum` - sum of all the values per aggregation window.
guid: "9ed41a78-3094-4e41-98b5-b1f98f63dc3a"	(Optional) A UUID unique for stat type. If missing, will be deduced based on `name` field.
},
. . .
],
name_map: "tags.instance"	The name of the input field to pick up a `node` name per `node type`.
}
],
skip_merge: true	The parser supports 2 modes of operations: merging all the fields in the input together and repartitioning again according to rules processing every input entry in the array separately Set this flag to `true` if every entry has to be parsed separately.
}

Eyer Knowledge base

Map your custom data source

Analytics

Requirements

Input data example (coming from Influx Telegraf)

Mapping example with explanation

Related content