Logs Ingestion - JSON

Endpoint: POST /api/{organization}/{stream}/_json

This will upload multiple records in batch with standard json format.

Response

{
	"code": 200,
	"status": [
		{
			"name": "stream1",
			"successful": 2,
			"failed": 0
		}
	]
}

Returns successful and failed count for each stream.

Restriction on number of fields/columns per record

Applicable to cloud version

Please note only records having 200 or less fields/columns will be considered for ingestion , records having more than 200 fields/columns will be discarded with failed status.

Applicable to open source version

One can configure ZO_COLS_PER_RECORD_LIMIT to set desired value for allowed number of fields/columns per record.

Flattening of the JSON structure

OpenObserve flattens deep JSON logs. Below is an example log before and after being flattened.

Before

{
	"actor": {
		"ip": "[redacted]",
		"id": 558875,
		"parent" : {
			"id": 45516,
			"active": true
		}
	}
	"response": {
		"error_occured": false,
		"status_code": 200
	}
	
}

After

{
	"actor_ip": "[redacted]",
	"actor_id": 558875,
	"actor_parent_id": 45516,
	"actor_parent_active": true,
	"response_error_occured": false,
	"response_status_code": 200
}

Restriction on flattening depth

⚠️ For performance reasons, OpenObserve limits the depth at which the JSON structure gets flattened. Past that limit, the generated field will contain unparsed JSON as a string. The default depth is 3, but this limit can be configured via the ZO_INGEST_FLATTEN_LEVEL environment variable. ZO_INGEST_FLATTEN_LEVEL can either be 0, which disables the flattening limit, or any positive number, to change the depth at which the flattening stops.

Timestamp

By default we add a field _timestamp for each record with the value of NOW in microseconds (unix epoch value).

we support use of two fields to override the default value.

_timestamp
@timestamp

2 data formats are supported for timestamp fields the value support two data type format:

microseconds (unix epoch value)
string value
- RFC 3339 and ISO 8601 date and time string such as 1996-12-19T16:39:57-08:00
- RFC 2822 date and time string such as Tue, 1 Jul 2003 10:52:37 +0200

eg:

use microseconds

[{
	"kubernetes.container_image": "quay.io/prometheus/prometheus:v2.39.1",
	"kubernetes.container_name": "prometheus", 
	"_timestamp": "1674789786006000"
}]

use string datetime

[{
	"kubernetes.container_image": "quay.io/prometheus/prometheus:v2.39.1",
	"kubernetes.container_name": "prometheus", 
	"_timestamp": "2023-01-02T10:01:01Z"
}]