Parsing data with Ingest node of Elastic Search

Please go through this link for ELK overview and explanation of each tool Elastic Stack Centralized logging solution practical explanation

There are 2 ways to parse the fields from log data

1. Shipping log data from file beats to logstash and use grok filters to parse the log line
2. Using Ingest Node of elastic search which preprocesses data before indexing

Second approach is easy one as filebeats are light weight shippers which are easy to handle and work

When you use Elasticsearch for output, you can configure Filebeat to use ingest node to pre-process documents before the actual indexing takes place in Elasticsearch. Ingest node is a convenient processing option when you want to do some extra processing on your data, but you do not require the full power of Logstash.

We use Grok Processors to extract structured fields out of a single text field within a document. A grok pattern is like a regular expression that supports aliased expressions that can be reused. Go through this blog on how to define grok processors you can use grok debugger to validate the grok patterns

Grok Processors in Elastic stack

 

Optionally you can add a data type conversion to your grok pattern. By default all semantics are saved as strings. If you wish to convert a semantic’s data type, for example change a string to an integer then suffix it with the target data type. For example %{NUMBER:num:int} which converts the num semantic from a string to an integer. Currently the only supported conversions are int and float.

So, if I add :int into my grok field definition, suddenly my value will be cast as an integer. Caveat: The NUMBER grok pattern will also detect numbers with decimal values. If you cast a number with a decimal as :int, it will truncate the decimal value leaving only the whole number portion.

In that case, the number can be cast as :float, meaning floating-point value.

One of the great things about Elasticsearch is its extensive REST API which allows you to integrate, manage and query the indexed data in countless different ways. We use Ingest Rest API of elasticsearch to put some pre processing rules on log data lines as below Here pipeline id = parse_logs (user defined name) but use the same name under elastic search configuration in filebeat.yml

The PUT pipeline API also instructs all ingest nodes to reload their in-memory representation of pipelines, so that pipeline changes take effect immediately. After defining the pipeline in Elasticsearch as above, we simply configure Filebeat to use the pipeline. To configure Filebeat, you specify the pipeline ID in the parameters option under elasticsearch in the filebeat.yml file:

output.elasticsearch:
  hosts: ["localhost:9200"]
  pipeline: parse_logs

sample message :

Here Field is message

“message”: “Home,103,1245,1005,1196,1343,4045,677,21840,0.00%,.3,21.4,2086.37”,

After creating pipeline we should able to see individual fields in kibana under discover page

This allows us to use advanced features like statistical analysis on value fields, faceted search, filters, and more.

Next blog is to analyse and visualize the parsed fields from the log data.

Advertisements

Elastic Stack Centralized logging solution practical explanation

Elastic Stack is the world’s most popular log management platform.

The ELK Stack is a collection of three open-source products — Elasticsearch,Logstash, and Kibana — all developed, managed and maintained by Elastic. Elasticsearch is a NoSQL database that is based on the Lucene search engine. Logstash is a log pipeline tool that accepts inputs from various sources, executes different transformations, and exports the data to various targets. Kibana is a visualization layer that works on top of Elasticsearch.

The stack also includes a family of log shippers(Lightweight Data Shippers) called Beats, which led Elastic to rename ELK as the Elastic Stack.

 

I will show you how to install Elastic search,Kibana and File Beats(Log file shippers) and leverage them to ship our daily logs to analyse and visualize meaning full insights.

Lot of ways to install them but for demo purpose i’m going with downloading Zip file in windows and later i will go through installing and running them in linux containers as well.

Infrastructure Setup:

• Download & Setup ElasticSearch
  • Ensure that you are able to access elastic search using http://ip-of-elasticsearch-host:9200
  • Run elasticsearch
• Download & Setup File Kibana
  • Update config/kibana.yml with the elastic url to fetch the data
  • Run kibana.bat / .sh
  • Ensure that you are able to access kibana using http://ip-of-kibana-host:5601
• Download & Setup File Beats
  • We need to have one file beat setup for each application instance we wish to ship the logs.
  • File Beat is responsible for collecting the log info and send it to elastic search engine.
  • Update filebeat.yml file as shown below

Just enable by  removing # before the line what ever you wish to have 

• Paths : file path where i kept jmeter.log using wildcard (.*)
• A Sample any log file format
• By default, File Beat logs each line in the log file as a separate log entry. Sometimes exceptions might span multi lines.  So we need to update filebeat.yml with the multiline pattern.
• For jmeter.log – each log entry has its time stamp. so, we can configure the pattern as multiline.pattern: ^[0-9]{4}/[0-9]{2}/[0-9]{2}(starting with time stamp)
• When there is no time stamp, FileBeat can append the line to the previous line based on the configuration.
• install Filebeat as service by running  (install-service-filebeat) powershell script under filebeat extracted folder so that it runs as a service and start collecting logs which we configured under path in yml file.
• FileBeat will start monitoring the log file – whenever the log file is updated, data will be sent to ElasticSearch. Using Kibana we can monitor the log entries in ElasticSearch.

Verify the filebeat logs under programdata/filebeat/logs folder and check to see filebeat started harvesting logs message as below

Login to kibana from the url http://ip-kibana-host/5601

So that you can ship all your logs to elastic search and from kibana UI we can visualize te data from elastic search.

In the next blog i will explain you how to parse the fields in the log message using grok filters and show how to create graphs  and dashboards out of log data.

 

Grok Processors in Elastic stack

What is Grok?

Data transformation and normalization in Logstash is performed using filter plugins. This article focuses on one of the most popular and useful filter plugins – Logstash Grok Filter, which is used to parse unstructured data into structured data making it ready for aggregation and analysis in the ELK.  This allows us to use advanced features like statistical analysis on value fields, faceted search, filters, and more.

Grok Basics

Grok sits on top of regular expressions, so any regular expressions are valid in grok as well. The regular expression library is Oniguruma, and you can see the full supported regexp syntax on the Onigiruma site.

Grok works by leveraging this regular expression language to allow naming existing patterns and combining them into more complex patterns that match your fields.

The syntax for reusing a grok pattern comes in three forms: %{SYNTAX:SEMANTIC}, %{SYNTAX}, %{SYNTAX:SEMANTIC:TYPE}.

Let’s take a look at some other available patterns (You can find a full list here).

The SYNTAX is the name of the pattern that will match your text. For example, 3.44 will be matched by the NUMBER pattern and 55.3.244.1 will be matched by the IP pattern. The syntax is how you match. NUMBER and IP are both patterns that are provided within the default patterns set.

The SEMANTIC is the identifier you give to the piece of text being matched. For example, 3.44 could be the duration of an event, so you could call it simply duration. Further, a string 55.3.244.1 might identify the client making a request.

The TYPE is the type you wish to cast your named field. int, long, double, float and boolean are supported types for coercion.

Here is an example of using the provided patterns to extract out and name structured fields from a string field in a document.

{
  "message": "55.3.244.1 GET /index.html 15824 0.043"
}

The pattern for this could be:

%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}

Here is an example pipeline for processing the above document by using Grok:

{
  "description" : "...",
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": ["%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}"]
      }
    }
  ]
}

This pipeline will insert these named captures as new fields within the document, like so:

{
  "message": "55.3.244.1 GET /index.html 15824 0.043",
  "client": "55.3.244.1",
  "method": "GET",
  "request": "/index.html",
  "bytes": 15824,
  "duration": "0.043"
}

Casting

Grok’ed fields are strings by default.  Numeric fields (int and float) can be declared in the pattern:

filter {
 grok {
   match => [ "message", "%{USERNAME:user:int}" ]
 }
}

Note that this is just a hint that logstash will pass along to elasticsearch when it tries to insert the event.  If the field already exists in the index with a different type, this won’t change the mapping in elasticsearch until a new index is created.

Debugging

There is an online grok debugger available for building and testing patterns.

It offers three fields:

1. The first field accepts one (or more) log line(s)
2. The second the grok pattern
3. The 3rd is the result of filtering the 1st by the 2nd

Convert JMeter JTL results to CSV or Graph format

Running JMeter scripts in Non GUI mode is easy and results can be available in JTL formats but the thing is in order to convert that to CSV , one thing is opening the JTL file from Jmeter GUI and saving the results to CSV but it requires manual effort.

In order to automate this process Jmeter-plugins has come up with command line utility to convert any JTL file data to Graphical representation and CSV format as well.

JMeterPluginsCMD

This small command-line utility for generating graphs out of JTL files. It behave just like right-click context menu on all graphs. The tool can generate PNG, CSV or both.

This tool can be downloaded from plugin manager which internally uses cmdrunner.jar to convert the required type of report.

Usage and Parameters¶

Generating PNG:

JMeterPluginsCMD.bat --generate-png test.png --input-jtl results.jtl --plugin-type ResponseTimesOverTime --width 800 --height 600

Generating CSV:

JMeterPluginsCMD.bat --generate-csv test.csv --input-jtl results.jtl --plugin-type ResponseTimesOverTime

Yoy may generate CSV and PNG in single tool run. --help will show you short help list on available parameters.

--generate-png <file>	generate PNG file containing graph
--generate-csv <file>	generate CSV file containing graph data
--input-jtl <file>	load data from specified JTL file
--plugin-type <class>	which type of graph use for results generation

Use --input-jtl merge-results.properties with --plugin-type MergeResults. The merge-results.properties file is in JMETER_HOME/bin dir.

if you open JMeterPluginsCMD.bat under jmeter installation directory/bin you would see the below Jar is being called internally when you run bat file

@echo off

java -jar %~dp0\..\lib\cmdrunner-2.2.jar –tool Reporter %*

so directly we can pass all the parameters to cmdrunner-2.2.jar as below

ECHO * * * * * GENERATE AGGREGATE CSV FILE * * * * * *
call java -jar C:\Projects\Performance\apache-jmeter-4.0\apache-jmeter-4.0\lib\cmdrunner-2.2.jar –tool Reporter –plugin-type AggregateReport –input-jtl sampleinput.jtl –generate-csv sampleoutput.csv

If you face any issues as below you may need to upgrade “synthesis” plugin: https://jmeter-plugins.org/?search=jpgc-synthesis to use the reporter from plugin manager.

Exception in thread “main” java.lang.RuntimeException: java.lang.ClassNotFoundException: kg.apc.jmeter.vizualizers.AggregateReportGui 

Sample bat file i use to run my scripts and save the output to Aggregate Report is as below

@ECHO OFF
REM
REM Set Output Directory and Filename
REM
Set Rpt_Dir=C:\Projects\Performance\JMeterScripts\Results
Set Out_file=ElasticBackendResults_%date:~10,4%%date:~4,2%%date:~7,2%
REM
REM CALL JMETER and run VL_UnifiedSuite test plan
REM
call C:\Projects\Performance\apache-jmeter-4.0\apache-jmeter-4.0\bin\jmeter -n -t C:\Projects\Performance\JMeterScripts\ElasticBackEndListerner.jmx -l %Rpt_Dir%\%Out_file%.jtl
REM
REM Convert JTL output into CSV file
REM
ECHO * * * * * GENERATE AGGREGATE CSV FILE * * * * * *
call java -jar C:\Projects\Performance\apache-jmeter-4.0\apache-jmeter-4.0\lib\cmdrunner-2.2.jar –tool Reporter –plugin-type AggregateReport –input-jtl %Rpt_Dir%\%Out_file%.jtl –generate-csv %Rpt_Dir%\%Out_file%.csv

Finally your results will be stored in CSV format.

Plugin Type Classes

Make note that you need to install corresponding plugins set to have Extras plugins available in CMD.

Most of class names are self-explanatory:

• AggregateReport = JMeter’s native Aggregate Report, can be saved only as CSV
• SynthesisReport = mix between JMeter’s native Summary Report and Aggregate Report, can be saved only as CSV
• ThreadsStateOverTime = Active Threads Over Time
• BytesThroughputOverTime
• HitsPerSecond
• LatenciesOverTime
• PerfMon = PerfMon Metrics Collector
• DbMon = DbMon Metrics Collector, DataBase, get performance counters via sql
• JMXMon = JMXMon Metrics Collector, Java Management Extensions counters
• ResponseCodesPerSecond
• ResponseTimesDistribution
• ResponseTimesOverTime
• ResponseTimesPercentiles
• ThroughputVsThreads
• TimesVsThreads = Response Times VS Threads
• TransactionsPerSecond
• PageDataExtractorOverTime

Ignore “Embedded Resouces” as error in JMeter

Sometimes we face our whole transactions to be failed even though only one of the non html resource was actually failed due to some reason.

example : org.apache.jmeter.protocol.http.parser.LinkExtractorParseException: java.lang.IllegalArgumentException: Unrecoverable error met during parsing, you can ignore such errors by setting property:’httpsampler.ignore_failed_embedded_resource’ to true,

In order to avoid this there are 2 steps one is just by enabling ignoring failed embedded resources setting in “Jmeter.Properties” file as below

One possibility is to edit ./bin/jmeter.properties and set httpsampler.ignore_failed_embedded_resources=true and uncomment to make it work.

If embedded resources download fails due to missing resources or other reasons, if this property is true. Parent sample will not be marked as failed.

Just restart the JMeter and run the script again to verify it

Getting Start and End dates of week/month/year in Apache JMeter

First have a look of Date and Time API for Java 8 as below

Java 8 Date and Time API

Below code helps to get the week start and end dates based on the current date.

import java.time.LocalDate;
import java.time.format.DateTimeFormatter;
import java.time.DayOfWeek;

LocalDate now=LocalDate.now(); //Gets the current date
DayOfWeek day=now.getDayOfWeek(); //Gets the day-of-week field, which is an enum DayOfWeek.
int value=day.getValue(); //Gets the day-of-week int value. as it starts from 0 which is monday
log.info(“value is “+value);
LocalDate weekStart=now.minusDays(value-1); // subtracting weekday value from the current date to find the weekstart date
LocalDate weekEnd=weekStart.plusDays(6); // adding value 6 to the weekstart gives us week end date as 6 is the enum constant value for Sunday
DateTimeFormatter formatter = DateTimeFormatter.ofPattern(“dd-MM-yyyy”); //formatting the date
String weekStartDate=weekStart.format(formatter); // converting date to string
String weekEndDate=weekEnd.format(formatter); // converting date to string
log.info(“value is “+weekStartDate+” “+weekEndDate);

 

2018-06-27 17:36:23,934 INFO o.a.j.e.JSR223PostProcessor: value is 3
2018-06-27 17:36:23,938 INFO o.a.j.e.JSR223PostProcessor: value is 25-06-2018 01-07-2018

As Current week start date is 25-06-2018 and end date should be 1-07-2018

Below code helps to get the week start and end dates based on the current date.

import java.time.*;
import java.time.format.*;

 

// Start and end dates of the month

String date = LocalDate.now().toString(); //Obtains the current date from the system clock in the default time-zone.
String monthStart= LocalDate.now().withDayOfMonth(1).toString(); //Returns a copy of this LocalDate with the day-of-month altered.
String monthEnd= LocalDate.now().withDayOfMonth(LocalDate.now().lengthOfMonth()).toString(); //Returns a copy of this LocalDate with the day-of-month altered.
log.info(“current date :”+date);
log.info(“monthStart date :”+monthStart);
log.info(“monthEnd date :”+monthEnd);

// Start and end dates of Year

String yearStart= LocalDate.now().withDayOfYear(1).toString(); //Returns a copy of this LocalDate with the day-of-month altered.
String yearEnd= LocalDate.now().withDayOfYear(LocalDate.now().lengthOfYear()).toString();

log.info(“current date :”+date);
log.info(“YearStart date :”+yearStart);
log.info(“YearEnd date :”+yearEnd);

2018-06-27 18:13:27,467 INFO o.a.j.m.JSR223PreProcessor: current date :2018-06-27
2018-06-27 18:13:27,467 INFO o.a.j.m.JSR223PreProcessor: monthStart date :2018-06-01
2018-06-27 18:13:27,467 INFO o.a.j.m.JSR223PreProcessor: monthEnd date :2018-06-30
2018-06-27 18:13:27,471 INFO o.a.j.m.JSR223PreProcessor: current date :2018-06-27
2018-06-27 18:13:27,471 INFO o.a.j.m.JSR223PreProcessor: YearStart date :2018-01-01
2018-06-27 18:13:27,471 INFO o.a.j.m.JSR223PreProcessor: YearEnd date :2018-12-31

 

// Combining all combinations in single code

import java.time.*;
import java.time.format.*;

LocalDate date = LocalDate.now(); //Obtains the current date from the system clock in the default time-zone.
LocalDate date1 =LocalDate.of(2018,05,18).minusMonths(1); //Obtains an instance of LocalDate from a year, month and day.
LocalDate firstdateofpreviousmonth = date.minusMonths(1).withDayOfMonth(date.lengthOfMonth()); //Returns a copy of this LocalDate with the day-of-month altered.
//LocalDate dateofpreviousmonth = date1.minusMonths(1)// Returns a copy of this LocalDate with the specified number of months subtracted.
// LocalDate firstdateofmonth = firstdateofpreviousmonth
LocalDate firstdateofyear = date.withDayOfYear(1); //Returns a copy of this LocalDate with the day-of-year altered.
DateTimeFormatter formatter = DateTimeFormatter.ofPattern(“yyyy-MM-dd”); //Datetime formatter
String monthdate = firstdateofpreviousmonth.format(formatter);
String setdate=date1.format(formatter);
String yeardate = firstdateofyear.format(formatter);
//String previousmonth = firstdateofpreviousmonth.format(formatter);
log.info(“date is ” + monthdate +” “+yeardate+” “+setdate);

String localDateTime = LocalDateTime.now().toString(); //Outputs this date as a String, such as 2007-12-03.
log.info(“Date in the format Year-Month-Day-Time: ” + localDateTime);

String localDateTime1 = LocalDateTime.now().format(DateTimeFormatter.ofPattern(“y-MM-dd HH:mm:ss”)); // gets the currentdate and Creates a formatter using the specified pattern.
log.info(“Date in the format Year-Month-Day-Time: ” + localDateTime1);

String localDateTime2 = LocalDateTime.now(ZoneId.of(“Europe/Paris”)).toString(); //Obtains the current date from the system clock in the specified time-zone and converts into date to a string
log.info(“Date for the Europe time zone: ” + localDateTime2);

String localDateTime3 = LocalDateTime.now(ZoneId.of(“UTC-05:00”)).toString(); //Obtains the current date from the system clock in the specified time-zone and converts into date to a string
log.info(“Date for the UTC time zone: ” + localDateTime3);

2018-06-27 18:20:46,826 INFO o.a.j.e.JSR223PostProcessor: date is 2018-05-30 2018-01-01 2018-04-18
2018-06-27 18:20:46,830 INFO o.a.j.e.JSR223PostProcessor: Date in the format Year-Month-Day-Time: 2018-06-27T18:20:46.830
2018-06-27 18:20:46,830 INFO o.a.j.e.JSR223PostProcessor: Date in the format Year-Month-Day-Time: 2018-06-27 18:20:46
2018-06-27 18:20:46,830 INFO o.a.j.e.JSR223PostProcessor: Date for the Europe time zone: 2018-06-27T14:50:46.830
2018-06-27 18:20:46,830 INFO o.a.j.e.JSR223PostProcessor: Date for the UTC time zone: 2018-06-27T07:50:46.830

If you follow all this scenarios you can deal with any kind of date , time and timezone scenarios

Handling Date and Time in Apache JMeter-Part-2

Working with dates in Java used to be hard. The old date library provided by JDK included only three classes: java.util.Date,java.util.Calendar and java.util.Timezone.

These were only suitable for the most basic tasks. For anything even remotely complex, the developers had to either use third-party libraries or write tons of custom code.

Another advantage is flexibility – working with multiple representations of time. The old date library included only a single time representation class – java.util.Date, which despite its name, is actually a timestamp. It only stores the number of milliseconds elapsed since the Unix epoch.

The new API has many different time representations, each suitable for different use cases:

• Instant – represents a point in time (timestamp)
• LocalDate – represents a date (year, month, day)
• LocalDateTime – same as LocalDate, but includes time with nanosecond precision
• OffsetDateTime – same as LocalDateTime, but with time zone offset
• LocalTime – time with nanosecond precision and without date information
• ZonedDateTime – same as OffsetDateTime, but includes a time zone ID
• OffsetLocalTime – same as LocalTime, but with time zone offset
• MonthDay – month and day, without year or time
• YearMonth – month and year, without day or time
• Duration – amount of time represented in seconds, minutes and hours. Has nanosecond precision
• Period – amount of time represented in days, months and years

I will give examples on LocalDate,LocalDateTime,ZonedDateTime,DateTimeFormatter usage

import java.time.*;
import java.time.format.*;

//Outputs this date as a String, such as ex 2007-12-03.

String localDateTime = LocalDateTime.now().toString(); 
log.info(“Date in the format Year-Month-Day-Time: ” + localDateTime);

// gets the currentdate and Creates a formatter using the specified pattern.

String localDateTime1 = LocalDateTime.now().format(DateTimeFormatter.ofPattern(“y-MM-dd HH:mm:ss”));
log.info(“Date in the format Year-Month-Day-Time: ” + localDateTime1);

String localDateTime2 = LocalDateTime.now(ZoneId.of(“Europe/Paris”)).toString(); //Obtains the current date from the system clock in the specified time-zone and converts into date to a string
log.info(“Date for the Europe time zone: ” + localDateTime2);

String localDateTime3 = LocalDateTime.now(ZoneId.of(“UTC-05:00”)).toString(); //Obtains the current date from the system clock in the specified time-zone and converts into date to a string
log.info(“Date for the UTC time zone: ” + localDateTime3);

 

 

dateNew = LocalDate.now (). minusDays (1) .toString ();

• LocalDate.now () – Creating the current date that is set on the computer on which the test is run
• minusDays (1) – minus 1 day from the current date. If it is necessary to take away, for example 10 days, then 1 must be replaced by 10.
• toString () – converting the created date into a variable with data type String and assigning the received value to the variable “dateNew”

Similarly, the rest of the usage variants work, depending on the value of the date variable in User Parameters.

Below are listed the actions that are performed by methods from the LocalDate class:

• LocalDate.now (). PlusYears (1) – creating the current date and adding one year to the current date
• LocalDate.now (). PlusMonths (1) – creating the current date and adding one month to the current date
• LocalDate.now (). PlusWeeks (1) – creating the current date and adding one week to the current date
• LocalDate.now (). MinusWeeks (1) – creating the current date and subtraction one week from the current date
• LocalDate.now (). MinusMonths (1) – creating the current date and subtraction one month from the current date
• LocalDate.now (). MinusYears (1) – creating the current date and subtraction one year from the current date
• vars.put (“dateNew”, dateNew); – creating a variable “dateNew”, and assigning the value of dateNew

 

code as below

//if you need specified date format you have to use DateTimeFormatter like this DateTimeFormatter.ofPattern(“y-MM-dd HH:mm:ss”)

 

import java.time.*;
import java.time.format.*;

String dateNew=LocalDate.now().toString();
log.info(“current date “+ dateNew);
String monthsBack=LocalDate.now().minusMonths(2).toString();
log.info(“2 months back date “+ monthsBack);
String monthsForward=LocalDate.now().plusMonths(3).toString();
log.info(“3 months forward date “+ monthsForward);
String weeksBack=LocalDate.now().minusWeeks(1).toString();
log.info(“1 week back date “+ weeksBack);
String daysForward=LocalDate.now().plusDays(2).toString();
log.info(“days forward date “+ daysForward);
String yearsBack=LocalDate.now().minusYears(2).toString();
log.info(“Years back date “+ yearsBack);
String setDate =LocalDate.of(2018,05,18).minusMonths(1).toString(); // set the date to custom and then moved back to 1 months
log.info(“Setdate and then get previous month date “+setDate);

Java 8 Date & Time API

https://docs.oracle.com/javase/8/docs/api/java/time/LocalDate.html

https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html

https://docs.oracle.com/javase/8/docs/api/java/time/LocalDateTime.html

https://docs.oracle.com/javase/8/docs/api/java/time/ZoneId.html