Cipher Keys

This page explains how to create and manage Cipher Keys in OpenObserve and how to use them to decrypt encrypted log data during search queries. The Cipher Keys feature is essential for handling sensitive data stored in encrypted formats while still enabling effective log search and analysis, without storing decrypted data on disk.

Create Cipher Keys

Follow these steps to create and configure Cipher Keys in OpenObserve:

Query Encrypted Logs with decrypt and decrypt_path

To retrieve original values from encrypted logs, use the decrypt() and decrypt_path() functions. These functions operate at query time and do not write decrypted data to disk.

decrypt(encrypted_field, 'cipher_key_name') 

SELECT
  decrypt(name_encrypted, 'customer_feedback_key') AS name
FROM "customer_feedback_encrypt"

SELECT
  decrypt(name_encrypted, 'customer_feedback_key') AS name,
  decrypt(feedback_encrypted, 'customer_feedback_key') AS feedback_text
FROM "customer_feedback_encrypt"

decrypt_path(encrypted_field, 'cipher_key_name', 'path')

{
  log: {
    "a": {
      "b": {
        "c": "ENCRYPTED_STRING"
      }
    }
  }
}

SELECT decrypt_path(log, 'my_key', 'a.b.c') AS decrypted_value FROM "encrypted_logs"

{
decrypted_value: xyz
}

SELECT
  decrypt_path(feedback_text, 'customer_feedback_key', '.') AS feedback_text_decrypted
FROM "customer_feedback_nested"

SELECT
  decrypt_path(metadata, 'customer_feedback_key', 'user.name') AS name
FROM "customer_feedback_nested"

{
"sessions": [
    { "id": "1", "token": "ENC(...)" },
    { "id": "2", "token": "ENC(...)" }
]
}

{
"data": [
    {
    "details": [
        { "secure": "ENC(...)" },
        { "secure": "ENC(...)" }
    ]
    }
]
}

Use decrypt and decrypt_path with Filter Clauses

You can use the decrypt() and decrypt_path() functions inside WHERE clauses to filter logs based on decrypted values. This enables querying on encrypted fields without storing or exposing the decrypted content.

• With the WHERE clause: The following query retrieves the name_encrypted from the customer_feedback_encrypt stream where the decrypted value of the name_encrypted field matches the string 'John Doe'. Note that the expected result is still in encrypted form.

Select name_encrypted from "customer_feedback_encrypt" where decrypt(name_encrypted,'customer_feedback_key') = 'John';

• With the LIKE clause: This query will decrypt the user_data field using the user_data_decryption_key and return results where the decrypted data contains the substring 'John':

SELECT user_id FROM user_activity_stream WHERE decrypt(user_data, 'user_data_decryption_key') LIKE '%John%';

Manage Cipher Keys

To update or delete a Cipher Key, click the edit or delete icon under the Actions column, respectively.

Cipher Key Storage in OpenObserve

When using OpenObserve as the encryption key (secret) storage option, you can encrypt the secret before storing it in the OpenObserve database by setting the following environment variables:

• O2_MASTER_ENCRYPTION_ALGORITHM: Set it to aes-256-siv (the default and the only supported value).
• O2_MASTER_ENCRYPTION_KEY: Set it to the encryption key you want OpenObserve to use to encrypt the secret. This must be a valid base64-encoded 512-bit aes-256-siv key. Tink Key cannot be used here.

Failure Modes

While managing Cipher Keys:

• Invalid Key Value During Update: If you provide invalid values while updating a Cipher Key, the system will display an error message and prevent the update from being saved. The existing Cipher Key value remains unchanged and valid. Ensure that the values you enter are correct.

• Duplicate Cipher Key Names: Ensure that each Cipher Key name is unique.

• Invalid Characters in Cipher Key Name: Ensure that Cipher Key names do not contain colons.

• Incorrect Encryption Key Type (Simple or Tink): When storing encryption keys, ensure you use the correct key type (Simple or Tink). If an incompatible key type is used, the system will show an error, and the key remains unsaved.

• Invalid Akeyless Credentials: If you enter invalid Akeyless credentials during Cipher Key creation or update, the system will reject the operation and show an error message. Correct the credentials and try again.

• Error in DFC Configuration:

  • For DFC without an Initialization Vector (IV), the system accepts any IV value except an empty one. An empty IV will trigger an error.
  • For DFC with an IV, the IV used for encryption must match exactly. A mismatch will result in an error during key creation or update.

While using decrypt function in Log search:

• Data is Not Encrypted (Plain Text): If the data is in plain text and not encrypted, the system will return the data as-is. Ensure that the data is encrypted if you intend to use the decrypt() function.
• Data Not Encrypted With the Given Encryption Key: If the data is not encrypted with the key you used to create the Cipher Key, the system will return the encrypted data as-is without decryption. Verify the encryption method and correct usage of the encryption key.
• No Permission on the Given Encryption Key: If you do not have permission to use the encryption key, the system will deny access and display an error message stating that you do not have permission. To resolve this, contact your administrator for access to the encryption key.

Limitation

• Cipher Key Usage (March 2025): Cipher Keys are currently used for decryption, and OpenObserve is working to expand their functionality to support encryption as well.
• decrypt() Function Scope (March 2025): As of now, the decrypt() function is limited to log search. OpenObserve is in the process of extending its capabilities to additional areas for broader application.