?

1 to N Juniper Log Parser

Overview

Receive log from Juniper and split the records into an array of entries.

In this example, real log from the Juniper which is in a string format is normalized into a standardized JSON format.

A Service (equvalent to a function in programming) will be used for this example.

Example Configuration

Learn how to use

Tutorial

Supporting Concepts

Basic concepts needed for the use case

Topic	Description
API	An API in API AutoFlow is simply an OpenAPI model
Server	A server accepts and handles the request and response.
Simulation	Data simulation is a mock data simulated for the purpose of visualizing the data in every step of the workflow. Simulated data is NOT the real data but a sample data you create. To use real data, use the Transaction feature to capture the data you send from Postman or CURL.
Scope	A scope is a namespace for variables.
Data Types	Data types describe the different types or kinds of data that you are gonna store and work with.

Use case specific concepts

Topic	Description
Action string/split-by	This Action returns array that divides a string into parts based on a pattern.
Action string/split-by-newline	This Action returns array that splits a given string based on a newline.
Action array/drop	This Action returns array that divides a string into parts based on a pattern.
Action iteration/map	Iterate over array of data mapping result to each array position.
Action string/split	This Action returns array that divides a string into parts based on a pattern.

Detail

The service has a single input:

• syslog: a string that has multiple lines of record.

last pid:  8197;  load averages:  0.65,  0.46,  0.42  up 21+12:45:04    22:32:33
107 processes: 4 running, 88 sleeping, 15 waiting

Mem: 163M Active, 90M Inact, 64M Wired, 72M Cache, 61M Buf, 92M Free
Swap:


  PID USERNAME    THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
   11 root          1 171   52     0K  3024K RUN    367.5H 68.75% idle
 8193 qatest        1 123    0 20300K 13364K RUN      0:01 34.34% cli
 8191 root          1 107    0  6648K  3040K select   0:00 12.81% sshd
 1208 root          1   8    0   135M 13640K nanslp  50.8H  1.03% pfem
 1205 root          1   4    0 80828K  9992K kqread 911:00  0.83% chassism
 1243 root          1  97    0  9120K  5676K select 100:30  0.05% license-check

The goal of the operation is to extract the IP address and port number from each record (each line).

The return should be an array of objects like:

[
  {
    "WCPU": "68.75%",
    "COMMAND": "idle"
  },
  {
    "WCPU": "34.34%",
    "COMMAND": "cli"
  }
  ...
]

Content

INPUT: HTTP Request

1: Create an API endpoint

Required Concepts

Learn how to create an API.

Create an API

From the left navigation, go to the API section and create a new API.

• ID: sample-data-transformation

Create an API Path

• Path: /1-to-n-juniper-parser
• Method: POST

2. Create a Server Operation

Required Concepts

Learn how to create a Server.

Create a Server

From the left navigation, go to the Server section and create a new Server.

• Server ID: sample-data-transformation
• Port Number: 1114 Feel free to select your own port number
• Linked API: sample-data-transformation (select the API you created above)

Create a Server Operation

• Press the "Add API Operation"
• Select the API endpoint created above

3 : Create Data Simulation using Real Data

Required Concepts

Learn how to create a Simulation.

We will use the "real data" to create the test simulation.

1. Send a HTTP request from Postman or cURL

API Autoflow Postman Collections

cURL

curl --location 'localhost:1114/1-to-n-juniper-parser' \
--header 'Content-Type: text/plain' \
--data 'last pid:  8197;  load averages:  0.65,  0.46,  0.42  up 21+12:45:04    22:32:33
107 processes: 4 running, 88 sleeping, 15 waiting

Mem: 163M Active, 90M Inact, 64M Wired, 72M Cache, 61M Buf, 92M Free
Swap:


  PID USERNAME    THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
   11 root          1 171   52     0K  3024K RUN    367.5H 68.75% idle
 8193 qatest        1 123    0 20300K 13364K RUN      0:01 34.34% cli
 8191 root          1 107    0  6648K  3040K select   0:00 12.81% sshd
 1208 root          1   8    0   135M 13640K nanslp  50.8H  1.03% pfem
 1205 root          1   4    0 80828K  9992K kqread 911:00  0.83% chassism
 1243 root          1  97    0  9120K  5676K select 100:30  0.05% license-check'

2. Check the data is received by the server endpoint

API Autoflow captures the data received and it can be used to create data simulation.

Action(s)

Required Concepts

Learn how to create a Actions.

Add actions to transform the data.

You can think of yourself as a cook. The big chunk of string you received is like a block of meat that you need to cut down into pieces for your recipe.

1. Split the document with the column header

The first big cut you are going to make is isolating the table. We can use String/split-by action to split with the table header PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND.

String Split By

SETTINGS

string: data

[request: body]

pattern: array

string PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND

OUTPUT

variables: output

2. Split the table in each line

Now that we have the table isolated, we can cut each line into a separate entry.

Note that original string was split into two pieces and we are working with the second piece stored in variable: output > 1.

String Split by New Line

SETTINGS

string: data

[variable: output > 1]

OUTPUT

variables: output

3. Remove the unwanted first line

During the split, the first line is created with an empty value. We can easily remove that by using Array drop action.

In the action setting, we are dropping 1 entry from the array that was created (output) from the previous action.

Array Drop

SETTINGS

array: data

[variable: output]

count: number 1

OUTPUT

variables: output

4. Remove the unwanted first line

• We will be iterating over the list of records that's saved in the scope variable: output.
  The previous action "string/split-by-newline" stored the output in the scope "variable: output".
• This is a special scope that exists only in the iteration. It is used to store the values as it iterates over the array.
  In our example, we will call it ip-port.

Iteration Map

Required Concepts

Learn how to use iteration/map actions.

SETTINGS

ITERATE: data

[variable: output]

SCOPE: string loop-var

String/Split: Split each value

As each line gets iterated, we can further split at every whitespace to isolate each value

SETTINGS

string: data

[loop-var: value]

OUTPUT

variables: output

RETURN: Create a NEW object

Now that we have isolated the data we need, we can create a NEW object to store the data.

SETTINGS

WCPU: data

[variable: output > 9]

COMMAND: data

[variable: output > 10]

OUTPUT

variables: output

OUTPUT: HTTP Response

1. Create a NEW object and map the IP and Subnet

Both the action's output and HTTP response body are set to variables: output. There's no change that needs to be made.

HTTP Response

SETTINGS

body: data

[variables: output]

Mapping the action output to the HTTP response output

• Data referenced in HTTP response is what gets sent back to the client.
• Map the output from the actions to be sent back.

NOTE: By default, the action output is set to variable output. If you intend to keep each action's output without it being overwritten by the next action, simply rename the output location in the action's output.

2. Test the API with Postman or CURL

cURL

curl --location 'localhost:1114/1-to-n-juniper-parser' \
--header 'Content-Type: text/plain' \
--data 'last pid:  8197;  load averages:  0.65,  0.46,  0.42  up 21+12:45:04    22:32:33
107 processes: 4 running, 88 sleeping, 15 waiting

Mem: 163M Active, 90M Inact, 64M Wired, 72M Cache, 61M Buf, 92M Free
Swap:


  PID USERNAME    THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
   11 root          1 171   52     0K  3024K RUN    367.5H 68.75% idle
 8193 qatest        1 123    0 20300K 13364K RUN      0:01 34.34% cli
 8191 root          1 107    0  6648K  3040K select   0:00 12.81% sshd
 1208 root          1   8    0   135M 13640K nanslp  50.8H  1.03% pfem
 1205 root          1   4    0 80828K  9992K kqread 911:00  0.83% chassism
 1243 root          1  97    0  9120K  5676K select 100:30  0.05% license-check'

---

Helpful?