ELK Stack Architecture 1

In this post we are going to look at an ELK stack architecture for a small scale implementation.  Keep in mind that this architecture is suitable for a small sized on-prem installation and the index capacity is determined by the hardware and disk space availability.

The following diagram shows the various components that were used for a specific use case to index about 30 GB of data a day with a 20 day retention requirement.

ELK stack architecture1

ELK stack beat’s agents are installed on servers to collect logs, metrics and network packet data. The collected data is then piped through logstash instances where the data is transformed,tags added, and then sent to elastic nodes to be indexed

Advertisements

   The diagram shows one instance of logstash for clarity purposes. In a production setup they would be multiple instances of logstash servers reading from beats agents and shipping to elastic instances.

    When setting up elastic nodes, there should be at-least 3 master nodes to avoid split brain scenario. This was previously discussed in Elastic Zen Discovery post. There are no limits on the number of data nodes that can be added to the cluster provided there are enough hardware resources. It is also recommended to setup elastic nodes that would help with indexing data using ingest nodes along with master and data nodes.

    Finally Kibana would be hosted in a separate server that is shown as Kibana in the diagram above. This can be single instance as Kibana works really well with even a single instance with multiple users at the same time.

The following is a general recommendation of hardware specs for each of the server nodes. The OS is Linux

  Logstash and Kibana nodes:

        Processor : 4 Core

        RAM           : 16 GB

        HDD           : 40GB

Elastic Data nodes:

      Processor : 8 Core

      RAM           : 24GB

      HDD           : 250 GB SSD

Elastic Master nodes:

      Processor : 4 Core

      RAM           : 16 GB

      HDD           : 120 GB SSD

NAS Storage:

       HDD        : 750 GB

More to follow in future articles.

Advertisements

Leave a Reply

Your email address will not be published. Required fields are marked *