Bulk PVLINE example

Hi,

This has been a busy week. As promised I will write something about Bulk collection and the pvline format. Please remember that you should always go back to the official documentation.

In TNPM, data can be collected using 2 modes: SNMP based or Bulk based. SNMP collection uses predefined formulas in TNPM (known as discovery / collection formulas) to query SNMP enabled devices and collect the necessary performance data. Bulk collection reads data from flat files.

Independent of the collection type used, the following steps will be necessary:

1. Discover the elements/subelements that you will collect data from
2. Collect the data for those elements/subelements using a predefined polling period
3. Present the data using reports and charts (you can also export the data)

If using SNMP, you will use SNMP discovery formulas to create the elements and subelements with its properties. If using BULK, all discovery information will be available in the flat file (that we are going to call "pvline" from now on).

Internally, TNPM will create for each BULK and each SNMP collector a kind of "processing line" called subchannel. For SNMP the first component in this processing line would be called something similar to SNMP.1.2 (channel 1, subchannel 2), and for BULK something like BCOL.2.3 (channel 2, subchannel 3).

The file is written in a format called PVLINE (full definition here). You can see an example below:

Type Both 
OPTION:Type=Line 
OPTION:PVMVersion=3.0 
OPTION:Element=NETD1
# Inventory Section
G1998/08/12 23:30:00 | Family | alias | NETD1_CPU1 |  | inventory | NETD_CPU
G1998/08/12 23:30:00 | Label | alias | NETD1_CPU1 |  | inventory | CPU C1
G1998/08/12 23:30:00 | Invariant | alias |  | NETD1_CPU1 | inventory | invcpu1
G1998/08/12 23:30:00 | Instance | alias | NETD1_CPU1 |  | inventory | cpu_<1>
G1998/08/12 23:30:00 | Slot | alias | NETD1_CPU1 |  | property | S1
G1998/08/12 23:30:00 | Frequency | alias | NETD1_CPU1 |  | property | 1GHz
(...)
OPTION:Element=NETD2
G1998/08/12 23:30:00 | Family | alias | NETD2_CPU1 |  | inventory | NETD_CPU
G1998/08/12 23:30:00 | Label | alias | NETD2_CPU1 |  | inventory | CPU C1
G1998/08/12 23:30:00 | Invariant | alias |  | NETD2_CPU1 | inventory | invcpu1
G1998/08/12 23:30:00 | Instance | alias | NETD2_CPU1 |  | inventory | cpu_<1>
G1998/08/12 23:30:00 | Slot | alias | NETD2_CPU1 |  | property | S1
G1998/08/12 23:30:00 | Frequency | alias | NETD2_CPU1 |  | property | 1GHz
(...)
# Data Section
G1998/08/12 23:30:00 | AP~Specific~Bulk~NETD_CPU~CPU_idle_pct | alias | NETD1_CPU1 |  | float | 25.00
G1998/08/12 23:30:00 | AP~Specific~Bulk~NETD_CPU~CPU_user_pct | alias | NETD1_CPU1 |  | float | 35.00
G1998/08/12 23:30:00 | AP~Specific~Bulk~NETD_CPU~CPU_system_pct | alias | NETD1_CPU1 |  | float | 40.00
(...)
G1998/08/12 23:30:00 | AP~Specific~Bulk~NETD_CPU~CPU_idle_pct | alias | NETD2_CPU1 |  | float | 35.00
G1998/08/12 23:30:00 | AP~Specific~Bulk~NETD_CPU~CPU_user_pct | alias | NETD2_CPU1 |  | float | 40.00
G1998/08/12 23:30:00 | AP~Specific~Bulk~NETD_CPU~CPU_system_pct | alias | NETD2_CPU1 |  | float | 25.00
(...)
G1998/08/12 23:45:00 | AP~Specific~Bulk~NETD_CPU~CPU_idle_pct | alias | NETD1_CPU1 |  | float | 10.00
G1998/08/12 23:45:00 | AP~Specific~Bulk~NETD_CPU~CPU_user_pct | alias | NETD1_CPU1 |  | float | 60.00
G1998/08/12 23:45:00 | AP~Specific~Bulk~NETD_CPU~CPU_system_pct | alias | NETD1_CPU1 |  | float | 30.00
(...)
G1998/08/12 23:45:00 | AP~Specific~Bulk~NETD_CPU~CPU_idle_pct | alias | NETD2_CPU1 |  | float | 20.00
G1998/08/12 23:45:00 | AP~Specific~Bulk~NETD_CPU~CPU_user_pct | alias | NETD2_CPU1 |  | float | 70.00
G1998/08/12 23:45:00 | AP~Specific~Bulk~NETD_CPU~CPU_system_pct | alias | NETD2_CPU1 |  | float | 10.00

When creating the pvline it is mandatory to add the ".pvline" extension in the filename (bfile.pvline for example).

Some important details about the pvline format (and common mistakes):

1. Please be careful with typos. Using "TYPE Both" can be different than "Type Both"
2. The file content must be in time sequence. The collector will ignore any line older than the last line read.
3. You can split the file into two main sections: The inventory section (lines with "inventory" or "property" in the example above) and the data section (lines with "float"). In the official documentation, you will see that the inventory and data sections are mixed. This is only useful if your inventory data for the same subelement will change inside the same file for different timestamps. If that is not the case, just write all inventory lines in the beginning of the file using the oldest timestamp ( "G1998/08/12 23:30:00" in our example) and then put all data after it.
4. The "OPTION:Element=" is only necessary in the inventory section. You don't have to put it in the data section
5. Please notice that the formula path does not contain "~" in the beginning. This is a very common error.

That is it. Once you have the BCOL running and you move the pvline file in its input directory, the file will be processed.

Talking about the file processing, some important points are listed below:

1. When using BULK collection, three facts must be true before any data can be stored in the database:
1. The subelement(s) must exist and be active
2. The collection formulas must exist and the formula requests must be configured and active in the RequestEditor
3. The oldest timestamp present in the file must be equal or newer than the last timestamp read by the BCOL. 
2. If configured to do the discovery, the BCOL will generate a new inventory file after every discovery time window (usually 60 min) of data processed. Please notice that the time window is related to the processed data, meaning that if your pvline file contains data from 09:00 until 12:30, it will generate 3 discovery files, one when processing the data for 10:00, another at 11:00 and another at 12:00.
3. The first processed hour for a new subelement is always discarded, once the subelement won't exist until the discovery window is reached and the inventory file created and processed.