Solution Architecture

Solution Architecture

Ubuntu Part

In order to connect an Elastic Cloud instance with FileBeats and Logstash, let’s first, install both in Ubuntu using the next commands:

sudo apt install filebeats logstash

FileBeat

We need to configure Filebeat to read the files from the server and ship the records to Logstash

An example configuration file is:

filebeat.inputs:
- type: filestream
  enabled: true
  paths:
    - /path/to/the/logfile.log

setup.template.settings:
  index.number_of_shards: 1

output.logstash:
  hosts: ["localhost:5043"]

logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0644

Logstash

We need to configure Logstash to get the receive the logs from FileBeats, parse them and send them to ElasticCloud.

An example configuration file is:

input {
    beats {
        port => "5043"
    }
}
filter {
    json {
        source => "message"
    }
}
output {
    elasticsearch {
    hosts => "https://***.europe-west3.gcp.cloud.es.io:9243"
    user => "elastic"
    password => "***"
    index => "myindex"
    document_type => "mylogs"
    }
}