How to ingest a JSON log file in Azure Log Analytics

This is a guide about ingesting a JSON log file into Azure Log Analytics Workspace

Linux Agent Install

First we install the Azure Log Analytics agent on the target linux computer:

If everything is ok, the we can check the agent health from Azure:

Input plugin configuration

Next we configure the configure input plugin by creating a file named log4j_json.conf in location /etc/opt/microsoft/omsagent/<workspace id>/conf/omsagent.d/

The content should be similar with the one bellow:

<source>
  @type tail
  path /path/to/logfile/log4j2-demo.log
  pos_file /tmp/log4j2-demo.log.pos
  tag oms.api.log4jjson
  format json
</source>

<match oms.api.log4jjson>
  type out_oms_api
  log_level info

  buffer_chunk_limit 5m
  buffer_type file
  buffer_path /var/opt/microsoft/omsagent/<workspace-id>/state/out_oms_api_log4jjson*.buffer
  buffer_queue_limit 10
  flush_interval 20s
  retry_limit 10
  retry_wait 30s
</match>

Please remember to change the ownership of the above file to the correct one:

sudo chown omsagent:omiusers /etc/opt/microsoft/omsagent/conf/omsagent.d/log4j_json.conf

Output plugin configuration

Now we link the input plugin with the output one by editing the file omsagent.conf in the location /etc/opt/microsoft/omsagent/<workspace id>/conf/omsagent.d/

Add the next configuration to the file:

<match oms.api.**>
  type out_oms_api
  log_level info

  buffer_chunk_limit 5m
  buffer_type file
  buffer_path /var/opt/microsoft/omsagent/<workspace-id>/state/out_oms_api*.buffer
  buffer_queue_limit 10
  flush_interval 20s
  retry_limit 10
  retry_wait 30s
</match>

Restart and wait

Now we restart the service

sudo /opt/microsoft/omsagent/bin/service_control restart

Now we should follow the log to check for errors:

tail -f /var/opt/microsoft/omsagent/log/omsagent.log

If everything is OK the log should look like the one bellow:

2022-07-01 17:23:12 +0000 [info]: Sending OMS Heartbeat succeeded at 2022-07-01T17:23:12.602Z
2022-07-01 17:24:12 +0000 [info]: Sending OMS Heartbeat succeeded at 2022-07-01T17:24:12.603Z
2022-07-01 17:25:12 +0000 [info]: Sending OMS Heartbeat succeeded at 2022-07-01T17:25:12.604Z
2022-07-01 17:26:12 +0000 [info]: Sending OMS Heartbeat succeeded at 2022-07-01T17:26:12.605Z
2022-07-01 17:26:20 +0000 [info]: OMS agent management service telemetry request success

Azure Log Analytics Custom Object

After a bit of time, the custom object and the data should appear in the Azure Log Analytics Workspace.

Documentation and links

• Install Log Analytics agent on Linux computers - https://docs.microsoft.com/en-us/azure/azure-monitor/agents/agent-linux
• Collecting custom JSON data sources with the Log Analytics agent for Linux in Azure Monitor - https://docs.microsoft.com/en-us/azure/azure-monitor/agents/data-sources-json
• Fluentd Tail plugin - https://docs.fluentd.org/input/tail