Friday, August 18, 2017

TNPM and RHEL 7 - Missing CSH library - Cannot Start Datachannel

Please notice that if you are using RHEL 7, CSH is not installed by default anymore. You have to install it to be able to start the Datachannel components.

yum install tcsh

The IBM doc only mentions KSH, but this is not enough.

Enjoy :)

Sunday, March 6, 2016

Installing TNPM 1.4.1 - Part 2 - What do you need?

Installation packages

Below you can find the installation packages and part numbers you will need:

IBM Software Catalog


CN5HGEN - IBM Tivoli Netcool Performance Manager Wireline Component v1.4.1 for Linux English

CIN3IML - IBM Tivoli Common Reporting 3.1.0.1 for Linux Multilingual

CIES6ML - IBM WebSphere Application Server V8.5.0.1 for Jazz for Service Management for Linux

CIXA2ML - Jazz for Service Management V1.1.0.3 for Linux Multilingual

CI6W6ML - IBM DB2 Enterprise Server Edition V10.1 for Linux on AMD64 and Intel® EM64T systems (x64) Multilingual

CI71NML - IBM DB2 10.1 Enterprise Server Edition - Restricted Use Quick Start and Activation Multiplatform Multilingual

FixCentral - IBM Installation Manager Install Kit for all x86_64 Linux versions supported by version 1.8.3.0

 

Oracle


p13390677_112040 - 64-bit Oracle Server 11g (11.2.0.4) Enterprise Edition for Linux

p13390677_112040 - 32-bit Oracle Client 11g (11.2.0.4) for Linux

System packages (RHEL 6.5)


# yum groupinstall "Base"

# yum install atk.i686 binutils.x86_64 cairo.i686 compat-libcap1.x86_64 compat-libstdc++-33.x86_64 compat-libstdc++-33.i686 elfutils-libelf.x86_64 elfutils-libelf-devel.x86_64 gcc.x86_64 gcc-c++.x86_64 glibc.x86_64 glibc.i686 glibc-common.x86_64 glibc-devel.x86_64 glibc-devel.i686 glibc-headers.x86_64 gtk2.i686 kernel-headers.x86_64 ksh.x86_64 libaio.x86_64 libaio.i686 libaio-devel.x86_64 libaio-devel.i686 libgcc.x86_64 libgcc.i686 libgomp.x86_64 libstdc++.x86_64 libstdc++.i686 libstdc++-devel.x86_64 libstdc++-devel.i686 libX11.i686 libXau.i686 libxcb.i686 libXext.i686 libXp.x86_64 libXp.i686 libXpm.x86_64 libXtst.i686 make.x86_64 openmotif.x86_64 openmotif.i686 openssl098e.i686 sysstat.x86_64 unixODBC.x86_64 unixODBC.i686 unixODBC-devel.x86_64 xorg-x11-xauth PackageKit-gtk-module.x86_64 libcanberra-gtk2.x86_64 gtk2-engines.x86_64 unzip openssh-clients.x86_64


Thursday, November 5, 2015

Installing TNPM 1.4.1 - Part 1 - Motivation

Every time I have to install TNPM the first thing I do is to ask myself: Do I have all I need? This looks a simple question, but if you consider that you need to download at least 10 different installation packs (yes 13GB), and match the compatible versions, this can take some time.

Another important question for me, especially when a new version is released: Are the installation instructions correct? Are they guiding me to the correct way of installing TNPM?

Unfortunately, this question proves to be more pertinent than I would like it to be. For instance, I already raised a very important issue when installing TNPM 1.3.2 about using the root user to install the TIP portal. TNPM 1.3.2 did that without fear and even finished with a chmod -R 777 <portal_home> in the end...very smart. IBM recognized the issue included some instructions on how to install the portal as non-root.

Well, how surprised I was, while reading the instructions for TNPM 1.4.1, to see that we have to install the portal (now called JAZZ) as the root user and do a chmod -R 777 <portal_home> as the post installation step. This is not necessary or even mandatory and absolutely not recommended for a public portal.

So, I hope the following posts will help you to do the it in a better way.

One important remark!!
Despite I will use only official IBM tools to install the different packages and no tricks, it always comes back to you to decide on the consequences of not following the official instructions (even if they are not good). It is your call, not mine, to make sure everything works on your environment. IBM support can eventually deny their support if they see your public portal is not running as root and all application files do not have the 777 permission.  

Monday, October 12, 2015

Launch TNPM pages in context from external applications

If you ever needed to integrate TNPM (v1.3.2+) with an external application, you may have asked yourself how to launch the TNPM portal in context with authentication and authorization in place.

It turns out there is a portal tool that allows exactly that. Is it called xlaunch and it is available for TIP and JAZZ portals. The tool will create a user-based token that can be used authenticate and launch the TNPM page you wanted using HTTP GET.

Steps:


1) Create the xlaunch credentials (all in one line):
 
$ java -cp /<JazzSM_HOME>/profile/installedApps/JazzSMNode01Cell/isc.ear/xlaunchapi.jar com.ibm.isc.api.xlaunch.LaunchPropertiesHelper\$Encode com.ibm.isc.xlaunch.username <username> com.ibm.isc.xlaunch.password <password>

Replace <username> and <password> with the TNPM user you want to SSO.

This will return a string like:

L2NvbS5pYm0uaXNjLnhsYXVuY2gujXNlcm5hbWUvdG5wbS1jb20uaWJtLmlzYy5KbGF1bmNoLnBhc3N3b3JkL3RucG0*

2) Launch in context (autologin using xlaunch credentials)
       
URL: https://<serverip>:16311/ibm/action/launch/<pageid>/<xlaunch_credential>

Example:
https://1.2.3.4:16311/ibm/action/launch/tnpm.console.topology.performance.network.resourceGroups/L2NvbS5pYm0uaXNjLnhsYXVuY2gujXNlcm5hbWUvdG5wbS1jb20uaWJtLmlzYy5KbGF1bmNoLnBhc3N3b3JkL3RucG0*

To get the pageid, open the page in JAZZ, click on the "single page" icon on the top right corner and click "About". Under "General" you will find the pageid

3) That's it

Wednesday, July 29, 2015

SevOne and TNPM. A technical comparison.

Some people have asked me about SevOne and how does it compare to TNPM. Some even ask if they should migrate from TNPM to SevOne. My answer is...it depends.

So, to help you take your decision, I prepared the table below. Please keep in mind that it is not my intention to say which one is better (I have my personal opinion though), but only to show what are the differences (and common) points between them and let you take your own decision.



Item IBM TNPM SevOne
Architecture 5 specialized core components:
  • Database
  • DataMart
  • DataView
  • DataChannel
  • DataLoad
1 core component:
  • SevOne appliance
Installation Very complex. Can take a long time (days to weeks) Very simple. Appliance or VM based. It is a matter of hours to install and start using
Upgrade Very complex. Can take a long time (days to weeks) Simple. Press the update button and it is done (complex architectures may demand some special care)
Scalability Via hardware upgrade on each core component Via cluster. Add another appliance side by side in cluster mode
SNMP collection Using collection formulas. Can be easily customized Using certified collection formulas. Usually needs SevOne intervention
BULK collection Using PVLINE (proprietary) format Using xStats component
Component Discovery Using discovery formulas. Can be easily customized and allow custom properties Not flexible. SevOne always discovers everything it can using SNMP walk. Allow custom properties
Component enrichment Using inventory hooks Using API
Licensing Based on “class of devices” being monitored. Very complex Per discovered and enabled object
Technical Portal WebSphere based. Allow some flexibility and customization Cannot be customized or changed
Services Portal Supported Not supported. Another solution needs to be used
Multi-tenancy Supported Supported
Reports Cannot be created by the user Can be created and shared by the user
Backup Database backup Not existent. To avoid data loss, another appliance needs to be paired in high availability
Cross collection Supported, but complex Supported
Grouping components (service like measurements) Supported, but complex. Can use rules to group components Supported but each component needs to be maintained manually or via the API
Historical data Limited by the database storage size. Purge scripts can be used. Usually limited to 1 year. This can be changed, but it is highly dependent on the local storage available on each appliance
Authentication LDAP or local LDAP, RADIUS, TACACS
Authorization Based on roles Based on roles
Data volume Report generation and data processing are subject to speed degradation as the data volume increases Report generation and data processing are NOT subject to speed degradation as the data volume increases
Data aggregation Pre-calculated and aggregated before stored in the database. Stored raw data and aggregated data side by side Only raw data is stored. Aggregated data is calculated during report rendering
Data export Can be done using DataAccess component Not trivial. API allows export of reports in CSV format, but not built for huge data export

Thursday, April 2, 2015

Error truncating partition

If for any reason you had an issue with your TNPM datachannel LDR that took more than 3 days to solve, you may find the following error on a LDR walkback file once it starts processing the backlog:

END OF WALKBACK20165: Error truncating partition
ORA-06512: at "PV_ADMIN.PVM_ERROR", line 137
ORA-06512: at "PV_ADMIN.PVM_DATALOAD", line 3558
ORA-06512: at line 1

This is caused due to the fact that the LDR is trying to do changes on a db partition that is already at the READONLY state. You can confirm it by searching the following on the oracle trace log:

$ORACLE_BASE/diag/rdbms/pv/pv/trace/proviso_PV_LDR_01.log
The Lowest Level Error Code is:
ORA-00372: file 487 cannot be modified at this time
ORA-01110: data file 487: '/(...)/PV_C01_1DGA_000_2014082500_001.dbf'

Unfortunately, the LDR will not proceed until the issue is solved and will keep generating walkbacks.

To solve the problem:

1) Connect to the oracle database as PV_ADMIN
2) Execute the query (replace the correct tablespace_name value as showed on the oracle trace log) :

select tablespace_name, status from dba_tablespaces where tablespace_name like '%C01_1DGA_000_2014082500%';

3) You should get two columns, one with the tablespace name and the other with "READ ONLY"

4) Change the tablespace to READ/WRITE executing the following (use the tablespace name returned by the previous sql query):

alter tablespace tablespace_name read write;

like:
alter tablespace C01_1DGA_000_2014082500 read write;

5) Bounce the LDR and it should work fine


Sunday, October 26, 2014

Piped merge error - what is wrong?

Last week a friend of mine came to me to ask about a strange error he was getting on TNPM. Basically, he had many gaps on report data for all devices, and it was apparently intermittent.

The error message on the log was the following:


V1:3353 2014.10.22-02.08.43 UTC LDR.1-21884:9897        SQLLDR  2 SQL Loader started
V1:3354 2014.10.22-02.08.43 UTC LDR.1-21884:13196       SQLLDR  3 Starting Piped Merge
V1:3355 2014.10.22-02.09.10 UTC LDR.1-21884:13196       MERGE_ERROR     GYMDC10118F Piped Merge Error: No such device or address:Transfer error
V1:3356 2014.10.22-02.09.10 UTC LDR.1-21884:13196       SQLLDRKILL      GYMDC10102W Killed sqlldr pid=a UnixProcess (Inactive: exitStatus nil, Error: Success) , result=a UnixProcess (Inactive: exitStatus nil, Error: Success)
V1:3357 2014.10.22-02.09.10 UTC LDR.1-21884:9897        ORASQLLDR       GYMDC10104F  ErrorCode=nil CommandLine=$ORA_HOME/11.2.0-client32/bin/sqlldr userid=PV_LDR_01/xxxx@pv log=...datachannel/LDR.1/state/2014.10.22-00/MERGED~000.1DGA.BOF.log control=...datachannel/LDR.1/loader.gagg.ctl logErrors=

After some investigation, it was evident that for some reason, the unix pipe created during the data merge was getting corrupted.

It is important to know that the LDR component has two options for merging and loading the data files. On the topology editor, if the option "USE_PIPE" is false, it will generate an intermediate file with the merged data and then use this file to upload via oracle sqlldr. If "USE_PIPE" is true, it will create a unix pipe and the oracle sqlldr will use it to load the files. Some people say that using the pipe is faster, because you don't have to create the intermediate file, but this can cause issues as well, as we will see.

The TNPM system I mentioned was using the pipe method for quite a long time before the issue occurred. So it, should be something in the system itself that had changed. And indeed, it was.

When using the pipe method, the pipe pointer is created under /tmp/LDR.X wher X is the LDR channel number. This works fine if the /tmp is mounted locally, but, if for any reason, the IT team decides to mount it using a remote data store... well, you will have problems. This was exactly what happened. The IT team decide to mount the /tmp using a remote data store for the VMware cluster. Once the datachannel was running on the cluster, the pipe load was affected.

So, we deactivated the pipe (USE_PIPE=false) on the topology editor, deployed the topology and the problem was solved.

I could not find a way to change where to create the pipe pointer, so it must be hard-coded somewhere. If you know how to do it, let me know :)

Sunday, October 19, 2014

link($DC_HOME/bin/visual,CMGR_visual) failed with error 18

If you ever installed the old version of TNPM (Proviso), you know that it was not possible to split the datachannel binaries from the data using two different locations. This was not very smart, considering that it is a common practice among many companies.

Since version 4.4.1 (I believe), this option exists and you can configure different locations for your datachannel binary files and the data files.

But, there is a catch if you use different partitions for the split (what is also the common practice...your data partition is usually remote mounted from the company data cluster).

The limitation is on the visual binary. You cannot execute it from a remote partition. If you try so, you will receive the following error on the screen:

link($DC_HOME/bin/visual,CMGR_visual) failed with error 18

This happens for all the tools that use the visual binary to bootstrap themselves (cmgr, amgr, frmi, etc...).

Fortunately, the solution is very simple:

1) Go to the $DC_HOME/bin folder
2) Open the run script used to start the tool. For instance "cmgr" for the CMGR component.
3) Go to the end of the script and add the line in red right above the last line:

cd $DC_BIN_HOME

$DC_BIN_HOME/pvexec CMGR_visual $DC_BIN_HOME/visual -nologo -noherald $DC_BIN_HOME/dc.im -headless -a CMGR "$@"

This will make sure you go back to the partition where your datachannel binaries are installed, before executing the visual command.

You have to do the same for all run scripts.

That's it.


Thursday, September 18, 2014

Datachannel component is not listed on "dccmd status all"

Suppose you cannot see BCOL.1.1 when you execute dccmd status all. Below, some things to investigate:

1) Try to run it manually. Run:
dccmd start BCOL.1.1

2) Check the proviso.log. Run:
grep "BCOL\.1\.1" proviso.log and search for any error message

3) If you see a WALKBACK error on the log, search for the walkback file and read the first lines to see the main reason for the walkback 

4) Check if the component configuration exists in the database. Run:
dccmd debug CMGR "self dbCfgPrint" | grep BCOL.1.1

This should show the component configuration in the database. Check for any mistakes and correct it using the topologyEditor

If you don't see any value, the component was not created on the topologyEditor or was not saved correctly in the database.

5) Check if any filesystem is full. Run:
df -kh and make sure no partition has 100% utilization

6) Restart some datachannel manager components (cmgr, amgr, cns) and try again.

If you still cannot find the reason, it is a good moment to open a ticket towards IBM 

Thursday, April 10, 2014

Tivoli Integrated Portal (TIP) portlet authorization

If you ever created custom pages in TIP, you may have encountered authorization issues. This is because you have to set the user authorization on different levels. The official documentation guides you on creating roles, groups, users and setting their relationships, but doesn't mention about portlet authorization.

Each page you create can include a portlet. The most common is the Web Widget, necessary to open a web page. By default, only the "administrator" role has access to it. So, if you create a page, add the web widget, and give access to non admin users to the page, when they try to access it, they will have a blank (grey) page and no error message displayed.

When that happens, go to the server log "SystemOut.log" and look for the following error:

"doStartTag() user does not have permissions for view mode"

Usually this is related to the portlet authorization. You will have to configure it as well.

To configure the portlet authorization, go to "Settings -> portlets" and click on the portlet. This will open the portlet configuration page. Click "Next" until you find the Security tab:


You can now add the correct roles to the authorization containers. Select "User" and add the role for you final user. Click "Next" and "Finish".

Now you can login again with the final user, open the page and the portlet will work fine.

Tuesday, March 25, 2014

TNPM 1.3.2 support for IE9 and Firefox ESR10

This link explains how to add support for Internet Explorer 9 and Firefox ESR10, but it is quite old.

I've just installed the last fixpacks for TCR and TIP and the steps I followed can be seen below:

Download the necessary packages

From IBM fixcentral download the following (please match the correct bit set for your installation):
  • Tivoli Netcool Performance Manager 1.3.2.0-TIV-TNPM-IF0042
  • Tivoli Integrated Portal 2.2.0-TIV-TIP-Linux64-FP0011
  • Tivoli Integrated Portal 2.2.0-TIV-TIP-FITSuit-FP00011
  • Tivoli Common Report 2.1.1.0-TIV-TCR-Linux64-FP2


Install in the following sequence (mandatory)


1) Tivoli Netcool Performance Manager 1.3.2.0-TIV-TNPM-IF0042

      Follow the instructions here

2) Tivoli Integrated Portal 2.2.0-TIV-TIP-Linux64-FP0011

      Follow the instructions on the 22011-fixpack-guide-PDF.pdf file (downloaded together with the packages from fixcentral)

3) Tivoli Common Report 2.1.1.0-TIV-TCR-Linux64-FP2

      Follow the instructions on the TCR211_FP2_Readme.txt file (downloaded together with the packages from fixcentral)

NOTE1: If for any reason you want to install using the console mode (Install.bin -i console), it still requires a valid DISPLAY set.

NOTE2: If for any reason you want to install using the silent mode (Install.bin -i silent -f <response_file>), you will not see any output at all on the shell, but the fix pack will run. You can follow up the process using the logs on <TCR_home>tipv2Components/TCRComponent/logs


Friday, February 7, 2014

Installing TNPM 1.3.2 with RHEL 5.9 Issues and Workarounds

Hi, I'm currently installing a fresh TNPM 1.3.2 platform and I believe some of the issues I found could be of interest. So, as I find them, I will list them here.


Oracle 32bit client MUST be installed on the 64bit server as well

Despite the documentation states the opposite ( see here ), it is mandatory to have the 32bit installed, otherwise, the topologyEditor and deployer won't be able to connect to the database. You MUST use the 32bit JDBC version when installing the topologyEditor and you MUST configure on the topologyEditor the ORACLE_HOME_ROOT property to point to your 64bit oracle install and ORACLE_HOME property to point to the 32bit client (and not the 64bit installed server) for you database host


Missing package for topologyEditor installation

If you try to install the topologyEditor and get this message:


"Graphical installers are not supported by the VM. The console mode will be used instead..."

It is because some 32bit library is missing. In my case, for RHEL 5.9 the missing 32bit libraries were libXres.i386 and libXtst.i386

Please notice that the same error doesn't occur for RHEL 5.5...


Missing packages for provisoinfo GUI on Datamart

If you see the following error message when opening the provisoinfod:

"java.lang.UnsatisfiedLinkError : awt (An exception was pending after running JNI_OnLoad)"

Please make sure all libX... packages have the 64bit and 32bit installed

# yum list installed | grep libX
libX11.i386                                1.0.3-11.el5_7.1            installed
libX11.x86_64                              1.0.3-11.el5_7.1            installed
libXTrap.i386                              1.0.0-3.1                   installed
libXTrap.x86_64                            1.0.0-3.1                   installed
libXau.i386                                1.0.1-3.1                   installed
libXau.x86_64                              1.0.1-3.1                   installed
libXaw.i386                                1.0.2-8.1                   installed
libXaw.x86_64                              1.0.2-8.1                   installed
libXcursor.i386                            1.1.7-1.2                   installed
libXcursor.x86_64                          1.1.7-1.2                   installed
libXdmcp.i386                              1.0.1-2.1                   installed
libXdmcp.x86_64                            1.0.1-2.1                   installed
libXext.i386                               1.0.1-2.1                   installed
libXext.x86_64                             1.0.1-2.1                   installed
libXfixes.i386                             4.0.1-2.1                   installed
libXfixes.x86_64                           4.0.1-2.1                   installed
libXfont.i386                              1.2.2-1.0.4.el5_7           installed
libXfont.x86_64                            1.2.2-1.0.4.el5_7           installed
libXfontcache.i386                         1.0.2-3.1                   installed
libXfontcache.x86_64                       1.0.2-3.1                   installed
libXft.i386                                2.1.10-1.1                  installed
libXft.x86_64                              2.1.10-1.1                  installed
libXi.i386                                 1.0.1-4.el5_4               installed
libXi.x86_64                               1.0.1-4.el5_4               installed
libXinerama.i386                           1.0.1-2.1                   installed
libXinerama.x86_64                         1.0.1-2.1                   installed
libXmu.i386                                1.0.2-5                     installed
libXmu.x86_64                              1.0.2-5                     installed
libXp.i386                                 1.0.0-8.1.el5               installed
libXp.x86_64                               1.0.0-8.1.el5               installed
libXpm.i386                                3.5.5-3                     installed
libXpm.x86_64                              3.5.5-3                     installed
libXrandr.i386                             1.1.1-3.3                   installed
libXrandr.x86_64                           1.1.1-3.3                   installed
libXrender.i386                            0.9.1-3.1                   installed
libXrender.x86_64                          0.9.1-3.1                   installed
libXres.i386                               1.0.1-3.1                   installed
libXres.x86_64                             1.0.1-3.1                   installed
libXt.i386                                 1.0.2-3.2.el5               installed
libXt.x86_64                               1.0.2-3.2.el5               installed
libXtst.i386                               1.0.1-3.1                   installed
libXtst.x86_64                             1.0.1-3.1                   installed
libXv.i386                                 1.0.1-4.1                   installed
libXv.x86_64                               1.0.1-4.1                   installed
libXxf86dga.i386                           1.0.1-3.1                   installed
libXxf86dga.x86_64                         1.0.1-3.1                   installed
libXxf86misc.i386                          1.0.1-3.1                   installed
libXxf86misc.x86_64                        1.0.1-3.1                   installed
libXxf86vm.i386                            1.0.1-3.1                   installed
libXxf86vm.x86_64                          1.0.1-3.1                   installed

Using oracle instance name different than PV

If you need to use another instance name, please make sure that you are defining EXACTLY the same value everywhere, specially the correct letter case. This is case-sensitive for the database instance creation!!

CheckDB script fails mentioning the os version is not supported

As you can see in the image below, the RHEL 5 updates 1 to 10 are supported by TNPM 1.3.2, but the checkdb script expects version 5.5U (hardcoded on checkdb.ini). Change the value in the checkdb.ini file and it should work.




Manual copy and execution of install packages

When installing TNPM, the deployer assumes you will enter the password for the application user on all remote servers and the root user. This is used to copy and execute the install package remotely. Now, this works fine when you have those passwords, but, in a real company environment, it will be unlikely you will receive them.

It is common to be able to do a "sudo" to the required application user and to transfer files using ssh shared keys among the servers (so no password needed).

Unfortunately, the deployer is not capable of transferring files remotely using ssh shared keys (this was identified as an APAR on Nov 2011...)

So, the solution is to disable the "remote file transfer" and "remote command execution" and do the work manually. The only problem is that there is another bug related to the install package that should be copied over (APAR opened on Jan 2012...)

Below you can find the steps necessary to manually create and execute the packages.

On the deployer disable the "remote file transfer" and the "remote execution". After, execute the installation plan step by step. It will give you an error for each step related to remote hosts. Once you see the error, right-click on the step and check the command output. You will see something mentioning you have to copy over the files under /tmp/ProvisoConsumer/(...)/runtime directory. Follow the instructions below for each step.

Deployer step: OS check

On the deployer host as root user:

  1. # cd /tmp/ProvisoConsumer/Plan/MachinePlan_<remote_host>/00001_CheckSys_OS_step_1_1 /runtime
  2. # mkdir package
  3. # cd package/
  4. # cp ../../bin/* .
  5. # cd ../../
  6. # tar cvf runtime.tar runtime/
On the remote host as application user:

  1. # mkdir -p /tmp/ProvisoConsumer/Plan/MachinePlan_<remote_host>/00001_CheckSys_OS_step_1_1/runtime
  2. # cd /tmp/ProvisoConsumer/Plan/MachinePlan_<remote_host>/00001_CheckSys_OS_step_1_1
  3. # scp <deployer_host>:/tmp/ProvisoConsumer/Plan/MachinePlan_<remote_host>/00001_CheckSys_OS_step_1_1/runtime.tar .
  4. # tar xvf runtime.tar
On the remote host as root user:

  1. # cd /tmp/ProvisoConsumer/Plan/MachinePlan_<remote_host>/00001_CheckSys_OS_step_1_1/runtime
  2. # ksh run.sh
If successfully executed (return code(s) = 0), go back to the deployer and change the step status as success for the "preparation" step and the "execution" step.

Deployer step: Database check

On the deployer host as root user:

  1. # cd /tmp/ProvisoConsumer/Plan/MachinePlan_<remote_host>/00002_CheckSys_DB_step_1_2/runtime
  2. # mkdir package
  3. # cd package/
  4. # cp ../../bin/* .
  5. # cd ../../
  6. # tar cvf runtime.tar runtime/
On the remote host as application user:

  1. # mkdir -p /tmp/ProvisoConsumer/Plan/MachinePlan_<remote_host>/00002_CheckSys_DB_step_1_2/runtime
  2. # cd /tmp/ProvisoConsumer/Plan/MachinePlan_<remote_host>/00002_CheckSys_DB_step_1_2
  3. # scp <deployer_host>:/tmp/ProvisoConsumer/Plan/MachinePlan_<remote_host>/00002_CheckSys_DB_step_1_2/runtime.tar .
  4. # tar xvf runtime.tar
On the remote host as root user:

  1. # cd /tmp/ProvisoConsumer/Plan/MachinePlan_<remote_host>/00002_CheckSys_DB_step_1_2/runtime
  2. # ksh run.sh
If successfully executed (return code(s) = 0), go back to the deployer and change the step status as success for the "preparation" step and the "execution" step.

Deployer step: JRE check

On the deployer host as root user:

  1. # cd /tmp/ProvisoConsumer/Plan/MachinePlan_<remote_host>/00003_JRE_step_2_1/runtime
  2. # mkdir package
  3. # cd package/
  4. # cp <tnpm_image>/v132/proviso/RHEL/jre/RHEL5/ibm\-java\-jre\-6.0\-8.1\-linux\-i386.tgz .
  5. # cd ../../
  6. # tar cvf runtime.tar runtime/
On the remote host as application user:

  1. # mkdir -p /tmp/ProvisoConsumer/Plan/MachinePlan_<remote_host>/00003_JRE_step_2_1/runtime
  2. # cd /tmp/ProvisoConsumer/Plan/MachinePlan_<remote_host>/00003_JRE_step_2_1
  3. # scp <deployer_host>:/tmp/ProvisoConsumer/Plan/MachinePlan_<remote_host>/00003_JRE_step_2_1/runtime.tar .
  4. # tar xvf runtime.tar
On the remote host as application user:

  1. # cd /tmp/ProvisoConsumer/Plan/MachinePlan_<remote_host>/00003_JRE_step_2_1/runtime
  2. # ksh run.sh
If successfully executed (return code(s) = 0), go back to the deployer and change the step status as success for the "preparation" step and the "execution" step.

Deployer step: Remote DataChannel setup

On the deployer host as root user:

  1. # cd /tmp/ProvisoConsumer/Plan/MachinePlan_<remote_host>/00004_DataChannel_step_8_1/runtime
  2. # mkdir package
  3. # cd package/
  4. # cp -R <tnpm_image>/v132/proviso/RHEL/DataChannel/* .
  5. # cd ../../
  6. # tar cvf runtime.tar runtime/
On the remote host as application user:

  1. # mkdir -p /tmp/ProvisoConsumer/Plan/MachinePlan_<remote_host>/00004_DataChannel_step_8_1/runtime
  2. # cd /tmp/ProvisoConsumer/Plan/MachinePlan_<remote_host>/00004_DataChannel_step_8_1
  3. # scp <deployer_host>:/tmp/ProvisoConsumer/Plan/MachinePlan_<remote_host>/00004_DataChannel_step_8_1/runtime.tar .
  4. # tar xvf runtime.tar
On the remote host as application user:

  1. # cd /tmp/ProvisoConsumer/Plan/MachinePlan_<remote_host>/00004_DataChannel_step_8_1/runtime
  2. # ksh run.sh
If successfully executed (return code(s) = 0), go back to the deployer and change the step status as success for the "preparation" step and the "execution" step.

Deployer step: Remote Datamart setup

On the deployer host as root user:

  1. # cd /tmp/ProvisoConsumer/Plan/MachinePlan_<remote_host>/00003_DataMart_step_7_1/runtime
  2. # mkdir package
  3. # cd package/
  4. # cp -R <tnpm_image>/v132/proviso/RHEL/DataMart/RHEL5/* .
  5. # cd ../../
  6. # tar cvf runtime.tar runtime/
On the remote host as application user:

  1. # mkdir -p /tmp/ProvisoConsumer/Plan/MachinePlan_<remote_host>/00003_DataMart_step_7_1/runtime
  2. # cd /tmp/ProvisoConsumer/Plan/MachinePlan_<remote_host>/00003_DataMart_step_7_1
  3. # scp <deployer_host>:/tmp/ProvisoConsumer/Plan/MachinePlan_<remote_host>/00003_DataMart_step_7_1/runtime.tar .
  4. # tar xvf runtime.tar
On the remote host as root user:

  1. # cd /tmp/ProvisoConsumer/Plan/MachinePlan_<remote_host>/00003_DataMart_step_7_1/runtime
  2. # ksh run.sh
If successfully executed (return code(s) = 0), go back to the deployer and change the step status as success for the "preparation" step and the "execution" step.

Deployer step: Remote TIP (TCR) setup

Please install TCR manually using its own installer and using the application username. After installing, mark this step as successful.

DO NOT install TCR using the deployer, once it will install the portal as root and as if that was not unsafe enough for an application portal (usually a public portal), it will execute chmod -R 777 <tip_home> as the last step. (yes, you can yell now...) 

Deployer step: Remote Dataview setup

On the deployer host as root user:

  1. # cd /tmp/ProvisoConsumer/Plan/MachinePlan_<remote_host>/00004_DataView_step_10_1/runtime
  2. # mkdir package
  3. # cd package/
  4. # cp -R <tnpm_image>/v132/proviso/RHEL/DataView/RHEL5/* .
  5. # cd ../../
  6. # tar cvf runtime.tar runtime/
On the remote host as application user:

  1. # mkdir -p /tmp/ProvisoConsumer/Plan/MachinePlan_<remote_host>/00004_DataView_step_10_1/runtime
  2. # cd /tmp/ProvisoConsumer/Plan/MachinePlan_<remote_host>/00004_DataView_step_10_1
  3. # scp <deployer_host>:/tmp/ProvisoConsumer/Plan/MachinePlan_<remote_host>/00004_DataView_step_10_1/runtime.tar .
  4. # tar xvf runtime.tar
On the remote host as application user:

  1. # cd /tmp/ProvisoConsumer/Plan/MachinePlan_<remote_host>/00004_DataView_step_10_1/runtime
  2. # ksh run.sh
If successfully executed (return code(s) = 0), go back to the deployer and change the step status as success for the "preparation" step and the "execution" step.

Deployer step: Remote Dataload setup

On the deployer host as root user:

  1. # cd /tmp/ProvisoConsumer/Plan/MachinePlan_<remote_host>/00003_DataLoad_step_9_1/runtime
  2. # mkdir package
  3. # cd package/
  4. # cp -R <tnpm_image>/v132/proviso/RHEL/DataLoad/* .
  5. # cd ../../
  6. # tar cvf runtime.tar runtime/
On the remote host as application user:

  1. # mkdir -p /tmp/ProvisoConsumer/Plan/MachinePlan_<remote_host>/00003_DataLoad_step_9_1/runtime
  2. # cd /tmp/ProvisoConsumer/Plan/MachinePlan_<remote_host>/00003_DataLoad_step_9_1
  3. # scp <deployer_host>:/tmp/ProvisoConsumer/Plan/MachinePlan_<remote_host>/00003_DataLoad_step_9_1/runtime.tar .
  4. # tar xvf runtime.tar
On the remote host as root user:

  1. # cd /tmp/ProvisoConsumer/Plan/MachinePlan_<remote_host>/00003_DataLoad_step_9_1/runtime
  2. # ksh run.sh
If successfully executed (return code(s) = 0), go back to the deployer and change the step status as success for the "preparation" step and the "execution" step.

Wednesday, September 4, 2013

No data in TNPM reports? Some things to investigate...

Hi,

This last week I saw something in TNPM that reminded me about one of the most common issues when processing and reporting performance data...no data in the reports. In this case it is quite obvious that something is wrong...but where is the problem?

I would like to share some steps I always take in order to investigate the problem in TNPM

Was data ever collected before?

This is the first thing to check...navigate to the past in the report (preferably a report with a chart in order to see raw data over time). Is there any datapoint available? If yes, when was the last one?
The most part of the time you will notice that the issue is not in TNPM itself, but in the data source. It can be a not announced patch applied to the device by the engineer dept (and they always forget to inform it :). It can be some network change caused by a new product release. It can be a change in the sftp credentials used to collect data...etc. Looking backward on time and identifying exactly the last collection point should be your first step.

Are the datachannel components running correctly?

Go to the Datachannel server and do a dccmd status all. Check if all components are on time or if one or many of them are delayed by some hours. Look for a column called "ES DURATION". The numbers indicate how long the component is in a "fixed state". Ideally you should see small numbers (let's say smaller than 100), indicating that something is being processed, except for the DISC component, that will usually have a big number.
Please be attentive here, specially if you are using CSE/CME formulas that depends on input from different subchannels. If one subchannel is delayed in time it will delay all dependent subchannels as well.

Are there ".bof" files being generated?

Go to the datachannel server and check the done directory for the subchannel with problems. You should see files with the extension ".bof" in it (except for the BCOL where you will find ".pvline" files). Please remember that the data flows in the following order:

BCOL
                => FTE   =>  CME   =>  LDR  => DLDR
SNMP

So, if you can find ".bof" files in the FTE/done directory and in the CME/done directory but not in the LDR/done directory for a specific subchannel, this means that the problem is in the CME and not in the LDR. 

Can you find some metric for your subelement in the ".bof" files?

Select one subelement that should have data displayed in the report, and get its dbIndex (the easiest way is to export the RST table to csv and look into the resource column).

Go to the data directory, find a recent ".bof" file and execute 

bofDump -r <dbIndex> <filename>.bof 

If nothing comes out, them the problem may be in the collection tree or collection requests (continue reading...)

Is the subelement in the correct folder in the collection tree?

Open the "pvm", go to the "resource editor" and check if the subelement is in the correct collection folder. If not, please check your grouping rules using the "rule editor".

Is a collection formula deployed and active for the collection folder?

Open the "request editor" and check if at least one collection formula is deployed and active for the collection folder.

NOTE: One small trick here. If the subelement exists in the collection folder, the formula is deployed and active and you can see data for other subelements in the same folder but not for this specific one, do a grep in the tnpm.log file using the dbIndex of this subelement and you will probably find an error saying the subelement was dropped because no request exists for it.

If this is the case, there is a problem with the CME local metadata image. Try the following:

1) Bounce the CME in question (dccmd bounce CME.X.Y)
2) If that doesn't solve, go the the "request editor", select the metrics deployed in the collection folder, disable them, save, enable them and save again. This was the only solution for me in some cases. I know it smells like a bug, but I don't have time right now to open a PMR (fell free to do it if you face this same problem :) )

Tuesday, January 29, 2013

Bulk PVLINE example

Hi,

This has been a busy week. As promised I will write something about Bulk collection and the pvline format. Please remember that you should always go back to the official documentation.

In TNPM, data can be collected using 2 modes: SNMP based or Bulk based. SNMP collection uses predefined formulas in TNPM (known as discovery / collection formulas) to query SNMP enabled devices and collect the necessary performance data. Bulk collection reads data from flat files.

Independent of the collection type used, the following steps will be necessary:
  1. Discover the elements/subelements that you will collect data from
  2. Collect the data for those elements/subelements using a predefined polling period
  3. Present the data using reports and charts (you can also export the data)
If using SNMP, you will use SNMP discovery formulas to create the elements and subelements with its properties. If using BULK, all discovery information will be available in the flat file (that we are going to call "pvline" from now on).

Internally, TNPM will create for each BULK and each SNMP collector a kind of "processing line" called subchannel. For SNMP the first component in this processing line would be called something similar to SNMP.1.2 (channel 1, subchannel 2), and for BULK something like BCOL.2.3 (channel 2, subchannel 3).

The file is written in a format called PVLINE (full definition here). You can see an example below:

Type Both 
OPTION:Type=Line 
OPTION:PVMVersion=3.0 
OPTION:Element=NETD1
# Inventory Section
G1998/08/12 23:30:00 | Family | alias | NETD1_CPU1 |  | inventory | NETD_CPU
G1998/08/12 23:30:00 | Label | alias | NETD1_CPU1 |  | inventory | CPU C1
G1998/08/12 23:30:00 | Invariant | alias |  | NETD1_CPU1 | inventory | invcpu1
G1998/08/12 23:30:00 | Instance | alias | NETD1_CPU1 |  | inventory | cpu_<1>
G1998/08/12 23:30:00 | Slot | alias | NETD1_CPU1 |  | property | S1
G1998/08/12 23:30:00 | Frequency | alias | NETD1_CPU1 |  | property | 1GHz
(...)
OPTION:Element=NETD2
G1998/08/12 23:30:00 | Family | alias | NETD2_CPU1 |  | inventory | NETD_CPU
G1998/08/12 23:30:00 | Label | alias | NETD2_CPU1 |  | inventory | CPU C1
G1998/08/12 23:30:00 | Invariant | alias |  | NETD2_CPU1 | inventory | invcpu1
G1998/08/12 23:30:00 | Instance | alias | NETD2_CPU1 |  | inventory | cpu_<1>
G1998/08/12 23:30:00 | Slot | alias | NETD2_CPU1 |  | property | S1
G1998/08/12 23:30:00 | Frequency | alias | NETD2_CPU1 |  | property | 1GHz
(...)
# Data Section
G1998/08/12 23:30:00 | AP~Specific~Bulk~NETD_CPU~CPU_idle_pct | alias | NETD1_CPU1 |  | float | 25.00
G1998/08/12 23:30:00 | AP~Specific~Bulk~NETD_CPU~CPU_user_pct | alias | NETD1_CPU1 |  | float | 35.00
G1998/08/12 23:30:00 | AP~Specific~Bulk~NETD_CPU~CPU_system_pct | alias | NETD1_CPU1 |  | float | 40.00
(...)
G1998/08/12 23:30:00 | AP~Specific~Bulk~NETD_CPU~CPU_idle_pct | alias | NETD2_CPU1 |  | float | 35.00
G1998/08/12 23:30:00 | AP~Specific~Bulk~NETD_CPU~CPU_user_pct | alias | NETD2_CPU1 |  | float | 40.00
G1998/08/12 23:30:00 | AP~Specific~Bulk~NETD_CPU~CPU_system_pct | alias | NETD2_CPU1 |  | float | 25.00
(...)
G1998/08/12 23:45:00 | AP~Specific~Bulk~NETD_CPU~CPU_idle_pct | alias | NETD1_CPU1 |  | float | 10.00
G1998/08/12 23:45:00 | AP~Specific~Bulk~NETD_CPU~CPU_user_pct | alias | NETD1_CPU1 |  | float | 60.00
G1998/08/12 23:45:00 | AP~Specific~Bulk~NETD_CPU~CPU_system_pct | alias | NETD1_CPU1 |  | float | 30.00
(...)
G1998/08/12 23:45:00 | AP~Specific~Bulk~NETD_CPU~CPU_idle_pct | alias | NETD2_CPU1 |  | float | 20.00
G1998/08/12 23:45:00 | AP~Specific~Bulk~NETD_CPU~CPU_user_pct | alias | NETD2_CPU1 |  | float | 70.00
G1998/08/12 23:45:00 | AP~Specific~Bulk~NETD_CPU~CPU_system_pct | alias | NETD2_CPU1 |  | float | 10.00

When creating the pvline it is mandatory to add the ".pvline" extension in the filename (bfile.pvline for example).

Some important details about the pvline format (and common mistakes):
  1. Please be careful with typos. Using "TYPE Both" can be different than "Type Both"
  2. The file content must be in time sequence. The collector will ignore any line older than the last line read.
  3. You can split the file into two main sections: The inventory section (lines with "inventory" or "property" in the example above) and the data section (lines with "float"). In the official documentation, you will see that the inventory and data sections are mixed. This is only useful if your inventory data for the same subelement will change inside the same file for different timestamps. If that is not the case, just write all inventory lines in the beginning of the file using the oldest timestamp ( "G1998/08/12 23:30:00" in our example) and then put all data after it.
  4. The "OPTION:Element=" is only necessary in the inventory section. You don't have to put it in the data section
  5. Please notice that the formula path does not contain "~" in the beginning. This is a very common error.

That is it. Once you have the BCOL running and you move the pvline file in its input directory, the file will be processed.

Talking about the file processing, some important points are listed below:
  1. When using BULK collection, three facts must be true before any data can be stored in the database:
    1. The subelement(s) must exist and be active
    2. The collection formulas must exist and the formula requests must be configured and active in the RequestEditor
    3. The oldest timestamp present in the file must be equal or newer than the last timestamp read by the BCOL. 
  2. If configured to do the discovery, the BCOL will generate a new inventory file after every discovery time window (usually 60 min) of data processed. Please notice that the time window is related to the processed data, meaning that if your pvline file contains data from 09:00 until 12:30, it will generate 3 discovery files, one when processing the data for 10:00, another at 11:00 and another at 12:00.
  3. The first processed hour for a new subelement is always discarded, once the subelement won't exist until the discovery window is reached and the inventory file created and processed.

Thursday, January 10, 2013

Introduction

Hi all,

This last week I've passed through some of my notes in order to decide from where to start. As I mentioned in my last post, the idea is not to repeat what is already written in the official documentation, so, if you want to know how to install the DataChannel or the minimum requirements, please check the official manuals (I know they are big...but you have to pass through it at least once :).

Before I continue, lets talk a little bit about history...

If you work with performance management for some time already, you probably heard about a product called Proviso...this is the former name of TNPM (or TNPM is its new name if you wish). In fact, after IBM acquired Quallaby, they decided to pick some other products from its portfolio and combine them under the same solution. They got Proviso, TIP, Cognos and Tivoli Netcool Performance Manager for Wireless, mixed them on the same cooking pot and produced two main dishes: TNPM wireline and TNPM wireless. In this blog we will discuss more about the wireline one.

Some other changes were done as well. IBM created two components for making the installation easier: one called TopologyEditor and another called Deployer. The idea was to centralize the configuration of all components in one place (the topology part) and allow local or remote installation from a central point (the deployer). Before, the configuration of each component was spread among many servers using config files. It was a good initiative from IBM, but I have to say that it brought with it more complexity to the process (and a lot of bugs as well).

Well, lets stop the history here...

As I mentioned earlier, I was checking my personal notes in order to find some interesting topics to write about and I found many of them:

  • Why does TNPM installs as root and is there a way to work around it?
  • How to manage users in TNPM?
  • Why SFTP doesn't work by default for bulk collections and how to force it?
  • Composite Subelements (CSE) introduction
  • Are you installing a Remote CME through a firewall? If you follow only the manual it won't work...
  • CME troubleshooting
  • BCOL troubleshooting
  • Creating Bulk adaptors, pvline format and other details
  • ...

I would like to try something here...instead of just choosing myself the topics sequence, I would like to ask you from where to start...The idea is to try to write about a different topic every week...I know this is a very new blog and probably few people know about it (you can change it sharing the link with your coworkers), but I want it to be as interactive as possible...so, please give me your opinion about which one you would like to see first or any suggestions you may have for new topics.

Au revoir!

Wednesday, January 2, 2013

And the story begins...

Hi all!!

Welcome to my new blog!

First of all, I would like to explain why I decided to create it. I've been working with IBM Tivoli Netcool Performance Manager (TNPM) for more than 3 years in different projects, installing, upgrading, fixing, and customizing it. I have to admit that it is a great product for performance management of network devices and highly scalable, but with a very complex installation and configuration process.

During those projects, I realized that despite the huge 100+ page manuals, the installation and configuration of each project required some extra information (the tips and tricks...)

So, that is the idea...share the knowledge I got with you. Of course I do not intend to replace the official manuals...they should be your first and main reference...but maybe you will find something here that will make your day (and mine as well).

That's it...let's start...