Changelog

What's Changed

Features

• feat: Add HTTP2 Streaming support (#6721)

• Adds SSE for _search
• SSE request will be fired for earch search request
• Results will be streamed as the partitions/cached are processed in the server with event progress.
• Streaming aggs support for SSE

Fixes

• fix: supercluster scan stats
• fix: ZO_DEFAULT_MAX_QUERY_RANGE_DAYS env also apply for date panel
• fix: scheduler query trace_id should be searchable in traces page
• fix: dashboard automation issue due to 4xx error msg change
• fix: internal error audit
• fix: consider actual records size per page for search job
• fix: search partition order_by
• fix: partition query format issue
• fix: parquet not found ingester
• fix: download manager improvements
• fix: validate download size with blob store and validate file
• fix: Update enrichment table cache with correct start time after meta table update and pipeline delayed timerange fix
• fix: simple distinct sql for http api
• fix: use real time id generator

New Env

ZO_STREAMING_ENABLED # default false
ZO_STREAMING_RESPONSE_CHUNK_SIZE_MB # default 1 mb
ZO_FILE_DOWNLOAD_ENABLE_PRIORITY_QUEUE # default true
ZO_FILE_DOWNLOAD_PRIORITY_QUEUE_THREAD_NUM # default 0
ZO_FILE_DOWNLOAD_PRIORITY_QUEUE_WINDOW_SECS # default 3600

New Metrics

zo_query_disk_cache_hit_count
zo_query_disk_cache_miss_count
zo_file_downloader_normal_queue_size
zo_file_downloader_priority_queue_size

HTTP Streaming output

With this feature we will use SSE to streaming output in one connection for each search request. We need to enable it by env first:

ZO_STREAMING_ENABLED=true

Then we can control it in the org setting page.