Tuesday, 28 April 2015

PLGC0049E: Propagation of the plug-in configuration file has failed for the Web server

If you get exceptions such as: -

PLGC0049E: Propagation of the plug-in configuration file has failed for the Web server

when trying to propagate the WebSphere Plugin configuration file from the WAS cell ( via the Deployment Manager ) to the IHS server, check that the Node Agent is running :-)

In my case, I have an unmanaged node ( so IHS is on the same "box" as the WAS Node Agent, and I am NOT running the IHS management server - apachectl ), and I'm using Jython to generate and propagate the Plugin configuration file ( plugin-cfg.xml ) and the TLS keyring: -

cellID=AdminControl.getCell()

AdminControl.invoke('WebSphere:name=PluginCfgGenerator,process=dmgr,platform=common,node=Dmgr,version=8.5.5.5,type=PluginCfgGenerator,mbeanIdentifier=PluginCfgGenerator,cell='+cellID+',spec=1.0', 'generate', '[/opt/IBM/WebSphere/AppServer/profiles/Dmgr01/config '+cellID+' Node1 webserver1 false]', '[java.lang.String java.lang.String java.lang.String java.lang.String java.lang.Boolean]')

AdminControl.invoke('WebSphere:name=PluginCfgGenerator,process=dmgr,platform=common,node=Dmgr,version=8.5.5.5,type=PluginCfgGenerator,mbeanIdentifier=PluginCfgGenerator,cell='+cellID+',spec=1.0', 'propagate', '[/opt/IBM/WebSphere/AppServer/profiles/Dmgr01/config '+cellID+' Node1 webserver1]', '[java.lang.String java.lang.String java.lang.String java.lang.String]')

AdminControl.invoke('WebSphere:name=PluginCfgGenerator,process=dmgr,platform=common,node=Dmgr ,version=8.5.5.5,type=PluginCfgGenerator,mbeanIdentifier=PluginCfgGenerator,cell='+cellID+',spec=1.0', 'propagateKeyring', '[/opt/IBM/WebSphere/AppServer/profiles/Dmgr01/config localhostCell01 ihsNode1 webserver1]', '[java.lang.String java.lang.String java.lang.String java.lang.String]')

For the record, these were the exceptions that I was seeing in the Deployment Manager's SystemOut.log: -

[28/04/15 11:43:06:194 BST] 000000cd SystemOut     O PLGC0062I: The plug-in configuration file is propagated from /opt/IBM/WebSphere/AppServer/profiles/Dmgr01/config/cells/PCCell1/nodes/Node1/servers/webserver1/plugin-cfg.xml to /opt/IBM/WebSphere/Plugins/config/webserver1/plugin-cfg.xml on the Web server computer.
[28/04/15 11:43:06:195 BST] 000000cd SystemOut     O PLGC0048I: The propagation of the plug-in configuration file is complete for the Web server. PCCell1.Node1.webserver1.
[28/04/15 11:43:06:195 BST] 000000cd SystemOut     O PLGC0063E: The propagation of the plug-in configuration file from /opt/IBM/WebSphere/AppServer/profiles/Dmgr01/config/cells/PCCell1/nodes/Node1/servers/webserver1/plugin-cfg.xml to /opt/IBM/WebSphere/Plugins/config/webserver1/plugin-cfg.xml on the Web server computer failed.
[28/04/15 11:43:06:195 BST] 000000cd SystemOut     O PLGC0049E: The propagation of the plug-in configuration file failed for the Web server. PCCell1.Node1.webserver1.
[28/04/15 11:43:06:196 BST] 000000cd MBeanHelper   E   Could not invoke an operation on object: WebSphere:name=PluginCfgGenerator,process=dmgr,platform=common,node=Dmgr,version=8.5.5.5,type=PluginCfgGenerator,mbeanIdentifier=PluginCfgGenerator,cell=PCCell1,spec=1.0 because of an mbean exception: com.ibm.websphere.plugincfg.exception.PluginConfigException: Applicaton centric propagation did not complete successfully: null
[28/04/15 11:50:31:123 BST] 000000cf SystemOut     O PLGC0062I: The plug-in configuration file is propagated from /opt/IBM/WebSphere/AppServer/profiles/Dmgr01/config/cells/PCCell1/nodes/Node1/servers/webserver1/plugin-cfg.xml to /opt/IBM/WebSphere/Plugins/config/webserver1/plugin-cfg.xml on the Web server computer.
[28/04/15 11:50:31:124 BST] 000000cf SystemOut     O PLGC0048I: The propagation of the plug-in configuration file is complete for the Web server. PCCell1.Node1.webserver1.
[28/04/15 11:50:31:127 BST] 000000cf SystemOut     O PLGC0063E: The propagation of the plug-in configuration file from /opt/IBM/WebSphere/AppServer/profiles/Dmgr01/config/cells/PCCell1/nodes/Node1/servers/webserver1/plugin-cfg.xml to /opt/IBM/WebSphere/Plugins/config/webserver1/plugin-cfg.xml on the Web server computer failed.
[28/04/15 11:50:31:127 BST] 000000cf SystemOut     O PLGC0049E: The propagation of the plug-in configuration file failed for the Web server. PCCell1.Node1.webserver1.
[28/04/15 11:50:31:129 BST] 000000cf MBeanHelper   E   Could not invoke an operation on object: WebSphere:name=PluginCfgGenerator,process=dmgr,platform=common,node=Dmgr,version=8.5.5.5,type=PluginCfgGenerator,mbeanIdentifier=PluginCfgGenerator,cell=PCCell1,spec=1.0 because of an mbean exception: com.ibm.websphere.plugincfg.exception.PluginConfigException: Applicaton centric propagation did not complete successfully: null

Saturday, 18 April 2015

The WebSphere Contrarian: Are you getting the most out of virtualization?

This popped up via IBM developerWorks on Twitter - @developerWorks: -

Much has happened in the area of virtualization since the topic was first discussed in this column, so this is a good time to revisit (or re-introduce) virtualization and all its flavors to help you determine which type of virtualization could benefit your organization the most, and how to make it the most effective. This content is part of the IBM WebSphere Developer Technical Journal.

Wednesday, 15 April 2015

How to manually generate a Heapdump in WebSphere on Windows

Problem(Abstract)

How to generate a heapdump without waiting for an OutOfMemoryError condition to occur.

Resolving the problem

Occasionally a heapdump needs to be generated without waiting for an OutOfMemoryError condition to occur. 

For example: an OutOfMemoryError will probably cause a system outage. On a production system this is not a desired event. If we manually create a heapdump before the OutOfMemoryError generates one, then the application server can be recycled and we will get a heapdump without an unscheduled outage. 

The manually generated heapdump will not be as useful as one which was generated by an OutOfMemoryError, but sometimes it can be of some use. The longer we can wait before generating the heapdump, the more likely the heapdump will contain the objects causing the OutOfMemory. If the heapdump is generated too soon, the largest memory user is not likely to be the same largest memory user when the heap is completely full.



IBM Business Monitor - Yet more on Cognos dependencies

This follows an earlier post: -


I saw a similar issue: -

Caused by: java.lang.UnsatisfiedLinkError: JCAM_Crypto_JNI (libCCLCore.so: cannot open shared object file: No such file or directory)

with Cognos 10.2.2 underlying IBM Business Monitor 8.5.6, during an installation on Red Hat Enterprise Linux 6.6 earlier today.

I went through the same problem determination regime as described in my earlier blog post, and found that the missing link was libstdc++.so.6.

Therefore, the solution was the same: -

yum install -y libstdc++.so.6 

which returned: -

...
Running Transaction
Installing : libstdc++-4.4.7-11.el6.i686 1/1 
Verifying : libstdc++-4.4.7-11.el6.i686 1/1 

Installed:
libstdc++.i686 0:4.4.7-11.el6 

Complete!

….

and all was then well.

IBM BPM - SCA and BPEL

This is something that I've recently added to my reading list.

It's by an IBM colleague, Mr Kim Clark Esq., and dates from 2010.

However, it's directly relevant to my current project, in the context of Long-Running and Straight Through business processing, hosted on IBM Business Process Manager Advanced 8.5.5.

I really really need to learn this stuff: -




and also this: -

Atomicity
Consistency
Isolation
Durability

Tuesday, 14 April 2015

IBM Business Monitor 8.5.6 on Red Hat Enterprise Linux 6.6 - Forked

Again, THIS is why I blog.

One of my colleagues saw issues with IBM Business Monitor 8.5.6 on a Red Hat Enterprise Linux 6.6 VM, including core dumps and OutOfMemory exceptions.

Given that I'd created the base VM, and also installed all of the IBM middleware components ( this is for an enablement event that we're co-delivering next week ), I wanted to get to the bottom of it.

I took his build notes, and went through the same process.

And I saw much the same thing, specifically: -

[4/14/15 20:06:37:172 BST] 00000001 ContainerHelp E   WSVR0102E: An error occurred stopping, com.ibm.ws.xs.httpsession.component.SessionListenerComponentImpl@907b28b4
java.lang.OutOfMemoryError: Failed to create a thread: retVal -1073741830, errno 11

[14/04/15 20:30:24:939 BST] 00000001 CommandMgr    E   ADMF0008E: Command Framework failed to initialize or cannot create CommandMgr in server mode. Root cause is java.lang.OutOfMemoryError: Failed to create a thread: retVal -1073741830, errno 11

in the AppClusterMember's SystemOut.log and: -

JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2015/04/14 20:29:55 - please wait.
JVMDUMP032I JVM requested System dump using '/opt/IBM/WebSphere/AppServer/profiles/BAMCell1AppSrv01/core.20150414.202955.3420.0001.dmp' in response to an event
JVMDUMP012E Error in System dump: insufficient system resources to generate dump, errno=11 "Resource temporarily unavailable"
JVMDUMP032I JVM requested Heap dump using '/opt/IBM/WebSphere/AppServer/profiles/BAMCell1AppSrv01/heapdump.20150414.202955.3420.0002.phd' in response to an event
JVMDUMP010I Heap dump written to /opt/IBM/WebSphere/AppServer/profiles/BAMCell1AppSrv01/heapdump.20150414.202955.3420.0002.phd
JVMDUMP032I JVM requested Java dump using '/opt/IBM/WebSphere/AppServer/profiles/BAMCell1AppSrv01/javacore.20150414.202955.3420.0003.txt' in response to an event
JVMDUMP010I Java dump written to /opt/IBM/WebSphere/AppServer/profiles/BAMCell1AppSrv01/javacore.20150414.202955.3420.0003.txt
JVMDUMP032I JVM requested Snap dump using '/opt/IBM/WebSphere/AppServer/profiles/BAMCell1AppSrv01/Snap.20150414.202955.3420.0004.trc' in response to an event
UTE001: Error starting trace thread for "Snap Dump Thread": -1
JVMDUMP010I Snap dump written to /opt/IBM/WebSphere/AppServer/profiles/BAMCell1AppSrv01/Snap.20150414.202955.3420.0004.trc


in the Node Agent's native_stderr.log.

Interestingly, I also saw this: -

tail -f SystemOut.log

-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
-bash: fork: retry: Resource temporarily unavailable
bash: fork: retry: Resource temporarily unavailable
-bash: fork: Resource temporarily unavailable

which immediately reminded me of not one BUT two blog posts from last year: -



at which point I thought …. ah, yes, about those ulimits :-)

I referred back to an older set of notes, and found this: -

Need to ensure ulimits  are set: -

Increase: -

open files                      (-n) 10240
max user processes              (-u) 1024


to: -

open files                      (-n) 65536
max user processes              (-u) 16384


as follows: -

Add: -

# - nofile - max number of open files
wasadmin   soft    nofile  65536
wasadmin   hard    nofile  65536

# - nproc - max number of processes
wasadmin   soft    nproc   16384
wasadmin   hard    nproc   16384

to: -

/etc/security/limits.d/90-nproc.conf

Once I did this, and rebooted, everything came up ( not smelling of roses, but working A-OK ), which is nice.

Guess what I've added to my VM template ready for next week ??

Sunday, 12 April 2015

Learn the workings of Git, not just the commands

This: -

Learn the workings of Git, not just the commands

When you come from using a classical source code repository like Concurrent Versions System (CVS) or Subversion (SVN), it can be difficult to understand the flexibility and the possibilities of Git. This article tries to explain the workings of Git and how it all fits together, so you don't have to try to remember the commands, but instead just know what to do because you understand it well.

via IBM developerWorks on Twitter.