Map your custom data source - Example 2
- Marius Wergeland Storsten
Eyer will work on any time-series data (metric + timestamp). You can connect open source agents like Influx Telegraf or Prometheus to support your technologies and applications, and you can also create custom agents / scripts that feed time-series data to Eyer.
To connect your custom data, you need to create a mapping from your source data to the Eyer internal format.
Requirements
Your data (from open source agent / custom agent / script) output must be in JSON format
Eyer currently only support metrics, not logs. If you work on logs, you can convert information of interest to metrics (e.g. number of errors matching a certain regex condition).
The metrics must have a timestamp in UNIX format (typically the time of metric collection)
The data must be mapped to the Eyer internal data structure (Node/metrics)
Input data example (coming from Influx Telegraf)
The example below uses Influx Telegraf to collect SNMP data (traffic in/out per port), and send them over HTTPS to the Eyer API. What you see below is the JSON sent from Influx Telegraf. The “fields” section can repeat, so a request can contain metrics from multiple ports.
Compared to mapping example 1, this mapping flattens all fields (metrics) into the same node, so there will be a single node with all port metrics.
{
"metrics":
[
{
"fields":
{
"ifHCInOctets": 0,
"ifHCOutOctets": 0
},
"name": "Octets_field",
"tags":
{
"agent_host": "192.168.88.54",
"host": "DESKTOP-S01F7CP",
"hostname": "US-8-150W",
"ifName": "3/26"
},
"timestamp": 1741167360
},
{
"fields":
{
"ifHCInOctets": 0,
"ifHCOutOctets": 0
},
"name": "Octets_field",
"tags":
{
"agent_host": "192.168.88.54",
"host": "DESKTOP-S01F7CP",
"hostname": "US-8-150W",
"ifName": "3/11"
},
"timestamp": 1741167360
},
{
"fields":
{
"ifHCInOctets": 0,
"ifHCOutOctets": 0
},
"name": "Octets_field",
"tags":
{
"agent_host": "192.168.88.54",
"host": "DESKTOP-S01F7CP",
"hostname": "US-8-150W",
"ifName": "3/19"
},
"timestamp": 1741167360
}
]
}Mapping example
We take the example above, and map it to the Eyer internal format. Below you see the mapping (in JSON). NOTE that this mapping place all metric fields in the same node based on the tags section “hostname” (US-8-150W)
{
"_id": {
"$oid": "67c17e6a2f90bb75049fb34b"
},
"selector": [
"tags.hostname",
"timestamp"
],
"type": "booomi.telegraf.snmp",
"id_selector": "tags.hostname",
"root": "metrics",
"system": "tags.hostname",
"skip_merge": true,
"mappings": [
{
"type": "booomi.telegraf.snmp.interface",
"name": "SNMP Device",
"guid": "7f92fc9c-23a3-462a-a556-9a7da1694e4a",
"timestamp": "timestamp",
"fields": [
{
"name": "fields.ifHCInOctets",
"name_map": "tags.ifName",
"display": "In Octets",
"type": "Count",
"aggregation": "avg",
"cumulative": true
},
{
"name": "fields.ifHCOutOctets",
"name_map": "tags.ifName",
"display": "Out Octets",
"type": "Count",
"aggregation": "avg",
"cumulative": true
}
],
"name_map": "tags.hostname"
}
]
}
Output data from the example data plus mapping
[
{
"Node":
{
"Name": "US-8-150W",
"SystemName": "US-8-150W",
"AgentId": "booomi.telegraf.snmp",
"AtomId": "US-8-150W"
},
"NodeType":
{
"Id": "7f92fc9c-23a3-462a-a556-9a7da1694e4a",
"Name": "booomi.telegraf.snmp.interface",
"DisplayName": "SNMP Device"
},
"Stats":
[
{
"Name": "fields.ifHCInOctets",
"Type": "Count",
"Guid": "66e12969-c225-5c6d-a5e1-8210488acb82",
"Aggregation": "avg",
"Display": "In Octets",
"Value": 0,
"Timestamp": 1741167360
},
{
"Name": "fields.ifHCOutOctets",
"Type": "Count",
"Guid": "66e12969-c225-5c6d-a5e1-8210488acb82",
"Aggregation": "avg",
"Display": "Out Octets",
"Value": 0,
"Timestamp": 1741167360
}
]
},
{
"Node":
{
"Name": "US-8-150W",
"SystemName": "US-8-150W",
"AgentId": "booomi.telegraf.snmp",
"AtomId": "US-8-150W"
},
"NodeType":
{
"Id": "7f92fc9c-23a3-462a-a556-9a7da1694e4a",
"Name": "booomi.telegraf.snmp.interface",
"DisplayName": "SNMP Device"
},
"Stats":
[
{
"Name": "fields.ifHCInOctets",
"Type": "Count",
"Guid": "66e12969-c225-5c6d-a5e1-8210488acb82",
"Aggregation": "avg",
"Display": "In Octets",
"Value": 0,
"Timestamp": 1741167360
},
{
"Name": "fields.ifHCOutOctets",
"Type": "Count",
"Guid": "66e12969-c225-5c6d-a5e1-8210488acb82",
"Aggregation": "avg",
"Display": "Out Octets",
"Value": 0,
"Timestamp": 1741167360
}
]
},
{
"Node":
{
"Name": "US-8-150W",
"SystemName": "US-8-150W",
"AgentId": "booomi.telegraf.snmp",
"AtomId": "US-8-150W"
},
"NodeType":
{
"Id": "7f92fc9c-23a3-462a-a556-9a7da1694e4a",
"Name": "booomi.telegraf.snmp.interface",
"DisplayName": "SNMP Device"
},
"Stats":
[
{
"Name": "fields.ifHCInOctets",
"Type": "Count",
"Guid": "66e12969-c225-5c6d-a5e1-8210488acb82",
"Aggregation": "avg",
"Display": "In Octets",
"Value": 0,
"Timestamp": 1741167360
},
{
"Name": "fields.ifHCOutOctets",
"Type": "Count",
"Guid": "66e12969-c225-5c6d-a5e1-8210488acb82",
"Aggregation": "avg",
"Display": "Out Octets",
"Value": 0,
"Timestamp": 1741167360
}
]
}
]