Wednesday, 1 July 2015

IBM BPM - Failing to uninstall

I had an interesting, and only half-resolved, issue with an Service Component Archive (SCA) module that I'd created using IBM Integration Designer 8.5.6 and deployed to the in-built IBM BPM Advanced 8.5.6 runtime ( known as the Unit Test Environment or Integrated Test Environment ).

I got into a position where I was unable to publish, start or uninstall an application ( merely a basic SCA module containing an Interface, a MQ Binding and a BPEL flow ).

This is what I saw in the WAS SystemOut.log file: -

...
[28/06/15 05:30:59:528 BST] 00000111 AdminHelper   A   ADMN1010I: An attempt is made to stop the BPMDemoApp application. (User ID = defaultWIMFileBasedRealm/wasadmin)
[28/06/15 05:30:59:532 BST] 00000111 CompositionUn A   WSVR0192I: Stopping composition unit WebSphere:cuname=BPMDemoApp in BLA WebSphere:blaname=BPMDemoApp.
[28/06/15 05:30:59:534 BST] 00000111 ApplicationMg A   WSVR0217I: Stopping application: BPMDemoApp
[28/06/15 05:30:59:986 BST] 00000111 Core          I   CWSCA3011I: The SCA module "BPMDemoApp" is stopping.
[28/06/15 05:30:59:990 BST] 00000111 ActivationSpe I   J2CA0524I: The Message Endpoint for ActivationSpec sca/BPMDemo/ActivationSpec (com.ibm.ws.sib.ra.inbound.impl.SibRaActivationSpecImpl) and MDB Application BPMDemoApp#BPMDemo#BPMDemo is deactivated.
[28/06/15 05:30:59:992 BST] 00000111 Core          I   CWSCA3012I: The SCA module has been stopped.
[28/06/15 05:31:00:007 BST] 00000111 ApplicationMg A   WSVR0220I: Application stopped: BPMDemoApp
[28/06/15 05:31:06:432 BST] 00000111 CompositionUn A   WSVR0193I: Composition unit WebSphere:cuname=BPMDemoApp in BLA WebSphere:blaname=BPMDemoApp stopped.
[28/06/15 05:31:06:838 BST] 00000113 ServletWrappe I com.ibm.ws.webcontainer.servlet.ServletWrapper init SRVE0242I: [isclite] [/ibm/console] [/secure/javascriptToSession.jsp]: Initialization successful.
[28/06/15 05:31:12:417 BST] 00000113 ServletWrappe I com.ibm.ws.webcontainer.servlet.ServletWrapper init SRVE0242I: [isclite] [/ibm/console] [/com.ibm.ws.console.appmanagement/RemoveSplat.jsp]: Initialization successful.
[28/06/15 05:31:15:971 BST] 00000125 FfdcProvider  W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on /opt/IBM/WebSphere/AppServer/profiles/bpmprofile/logs/ffdc/server1_267904e7_15.06.28_05.31.15.9676531005148371738678.txt com.ibm.ws.management.application.ExtensionHelper.processServerUninstallExtensions 290
[28/06/15 05:31:15:983 BST] 00000125 FfdcProvider  W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on /opt/IBM/WebSphere/AppServer/profiles/bpmprofile/logs/ffdc/server1_267904e7_15.06.28_05.31.15.971705321591203166738.txt com.ibm.ws.management.application.SchedulerImpl.run 328
[28/06/15 05:31:15:985 BST] 00000125 UninstallSche I   ADMA5107E: The application BPMDemoApp cannot be uninstalled.

...

with this in the FFDC logs: -

...
[28/06/15 05:31:15:968 BST]     FFDC Exception:java.lang.ClassCastException SourceId:com.ibm.ws.management.application.ExtensionHelper.processServerUninstallExtensions ProbeId:290 Reporter:java.lang.Class@2bc12930
java.lang.ClassCastException: com.ibm.ws.management.application.UninstallSchedulerImpl incompatible with com.ibm.websphere.management.application.InstallScheduler
at com.ibm.task.admin.HumanTaskTaskProvider.provideServerUninstallExtensions(HumanTaskTaskProvider.java:187)
at com.ibm.ws.management.application.ExtensionHelper.processServerUninstallExtensions(ExtensionHelper.java:286)
at com.ibm.ws.management.application.UninstallSchedulerImpl.setupTasks(UninstallSchedulerImpl.java:156)
at com.ibm.ws.management.application.SchedulerImpl.run(SchedulerImpl.java:303)
at java.lang.Thread.run(Thread.java:801)

...
[28/06/15 05:31:15:971 BST]     FFDC Exception:com.ibm.websphere.management.application.client.AppDeploymentException SourceId:com.ibm.ws.management.application.SchedulerImpl.run ProbeId:328 Reporter:java.lang.Class@2bc12930
com.ibm.websphere.management.application.client.AppDeploymentException: java.lang.ClassCastException: com.ibm.ws.management.application.UninstallSchedulerImpl incompatible with com.ibm.websphere.management.application.InstallScheduler [Root exception is java.lang.ClassCastException: com.ibm.ws.management.application.UninstallSchedulerImpl incompatible with com.ibm.websphere.management.application.InstallScheduler]
        at com.ibm.ws.management.application.ExtensionHelper.processServerUninstallExtensions(ExtensionHelper.java:294)
        at com.ibm.ws.management.application.UninstallSchedulerImpl.setupTasks(UninstallSchedulerImpl.java:156)
        at com.ibm.ws.management.application.SchedulerImpl.run(SchedulerImpl.java:303)
        at java.lang.Thread.run(Thread.java:801)
Caused by: java.lang.ClassCastException: com.ibm.ws.management.application.UninstallSchedulerImpl incompatible with com.ibm.websphere.management.application.InstallScheduler
        at com.ibm.task.admin.HumanTaskTaskProvider.provideServerUninstallExtensions(HumanTaskTaskProvider.java:187)
        at com.ibm.ws.management.application.ExtensionHelper.processServerUninstallExtensions(ExtensionHelper.java:286)
        ... 3 more
...


I then tried to uninstall the application using an old-but-good Jacl script: -

/opt/IBM/WebSphere/AppServer/profiles/bpmprofile/bin/wsadmin.sh -lang jacl -user wasadmin -password passw0rd -f /opt/IBM/WebSphere/AppServer/ProcessChoreographer/admin/bpcTemplates.jacl -uninstall BPMDemoApp -force

WASX7209I: Connected to process "server1" on node Node1 using SOAP connector;  The type of process is: UnManagedProcess
WASX7303I: The following options are passed to the scripting environment and are available as arguments that are stored in the argv variable: "[-uninstall, BPMDemoApp, -force]"

Script 'bpcTemplates.jacl' has been deprecated in BPM 8.5.5 and replaced by /opt/IBM/WebSphere/AppServer/ProcessChoreographer/admin/listTemplates.py and manageTemplates.py


Application 'BPMDemoApp' is deployed on server 'server1' of node 'Node1'.
   Process templates:
     No process templates found in 'BPMDemoApp'.
   Task templates:
     No task templates found in 'BPMDemoApp'.
ADMA5011I: The cleanup of the temp directory for application BPMDemoApp is complete.
ADMA5107E: The application BPMDemoApp cannot be uninstalled.
Error stopping and force uninstalling BPMDemoApp: com.ibm.ws.scripting.ScriptingException: WASX7133E: Application uninstall of BPMDemoApp failed: see previous messages for details..


which struck out :-(

<CAVEAT> Don't try this at home, kids</CAVEAT>

Finally, I took the "nuclear" option of physically removing the EAR file: -

cd /opt/IBM/WebSphere/AppServer/profiles/bpmprofile/config/cells/PSCell1/applications
rm -Rf BPMDemoApp.ear/


cd /opt/IBM/WebSphere/AppServer/profiles/bpmprofile/installedApps/PSCell1
rm -Rf BPMDemoApp.ear/


and validated that, at least from a file-system perspective, it had gone: -

find /opt/IBM/WebSphere/AppServer/ -name BPMDemo

which returned NOWT.

<CAVEAT> Don't try this at home, kids</CAVEAT>

The application is still "there" as far as WAS is concerned, as it appears in the Applications perspective in the ISC, but it's not stopping my server from starting, so I can ignore it - for now.

And this is only my personal test VM.

Monday, 29 June 2015

IBM Business Process Manager concepts and terminology

Some useful background about IBM Business Process Manager .. from the IBM Integration Bus documentation: -

IBM Business Process Manager concepts and terminology

You can become more familiar with Business Process Manager (BPM) concepts to clarify how it interacts with IBM® Integration Bus.

What is Process Designer?

IBM Process Designer is a graphical user interface tool. You can use it to quickly model and implement your business processes. You then test and demonstrate the process design during the development phase. For more information about Process Designer, look for getting started with Process Designer in the Business Process Manager product documentation.

What is a Process Center?

You use the tools and repository of a Process Center to model the processes, services, and other assets in a business process by using IBM Process Designer. For more information about Process Center, look for process designer and process center tasks in the Business Process Manager product documentation.

What are business process definitions?

A business process definition is a model, you create in Process Designer, of a business process. For more information, look for business process definitions (BPDs) in the Business Process Manager product documentation.

What are business objects?

A business object is a set of fields or elements that you use together to represent a meaningful business process. You define business objects as variables to pass information through a business process. Each business object can be a primitive data type (such as, a string or an integer) or it can itself be a business object. For more information, look for business objects in the Business Process Manager product documentation.

What are system tasks and Integration Services?

System tasks are activities in a business process definition that are completed by an automated system. For more information, look for understanding task types in the BPM information centre.

An Integration Service is a particular type of system task that you use to access the web and other services. For more information, look for understanding service types in the Business Process Manager product documentation.

What are toolkits?

A BPM toolkit is a library of process designer artifacts; such as business objects, Integration Services, or business process definitions. A toolkit can be used for sharing between business process definitions. Toolkits are stored and managed with Process Center. For more information about toolkits, look for managing toolkits in the Business Process Manager product documentation.

What are snapshots?

You make versions by using snapshots. A snapshot records the state of library items (such as toolkits) at a specific time. For more information, look for creating snapshots in the process center console in the Business Process Manager product documentation.

What is a BPM export .twx file?

You import an export .twx file to transport BPM artifacts between Process Centers or to integrate with IBM Integration Bus. The format for the export .twx file is proprietary. For more information, look for importing and exporting process applications from the process center console in the Business Process Manager product documentation.

Sunday, 28 June 2015

Whilst I remember - parsing my JSON in IBM Integration Bus

Following my earlier post: -


I'm writing this down before I forget.

Previously,I'd noticed that my Compute Node was returning JSON in this format: -

message <{"Employee":{"row":{"EMPNO":"000100","FIRSTNME":"THEODORE","LASTNAME":"SPENSER"}}}>

which didn't really tie up with my expectations, given that the DB2 schema is like this: -

Employee
EMPNO
FIRSTNME
LASTNAME

I fiddled around with my ESQL, changing it from this: -

SET OutputRoot.JSON.Data.Employee = rowRef;

to: -

SET OutputRoot.JSON.Data.Employee = dbResultSetRef.row;

In other words, I removed the reference to rowRef and instead replaced it with dbResultSetRef.row.

I now see this: -

message <{"Employee":{"EMPNO":"000100","FIRSTNME":"THEODORE","LASTNAME":"SPENSER"}}>
message <{"Employee":{"EMPNO":"000200","FIRSTNME":"DAVID","LASTNAME":"BROWN"}}>
message <{"Employee":{"EMPNO":"000300","FIRSTNME":"PHILIP","LASTNAME":"SMITH"}}>

from input such as this: -

{"EmployeeID":"000100"}
{"EmployeeID":"000200"}
{"EmployeeID":"000300"}


I also learned about the Trace Node



which returns some tasty debug: -

Jun 28 20:37:54 bpmdemo IIB[77236]: IBM Integration Bus v9002 (IB9NODE.IIB9) [Thread 90570] (Msg 1/1) BIP3051E: Error message '( ['MQROOT' : 0x7f68b80d3e00]   (0x01000000:Name  ):Properties = ( ['MQPROPERTYPARSER' : 0x7f68b803c390]     (0x03000000:NameValue):MessageSet             = NULL     (0x03000000:NameValue):MessageType            = NULL     (0x03000000:NameValue):MessageFormat          = NULL     (0x03000000:NameValue):Encoding               = NULL     (0x03000000:NameValue):CodedCharSetId         = NULL     (0x03000000:NameValue):Transactional          = NULL     (0x03000000:NameValue):Persistence            = NULL     (0x03000000:NameValue):CreationTime           = NULL     (0x03000000:NameValue):ExpirationTime         = NULL     (0x03000000:NameValue):Priority               = NULL     (0x03000000:NameValue):ReplyIdentifier        = NULL     (0x03000000:NameValue):ReplyProtocol          = 'MQ' (CHARACTER)     (0x03000000:NameValue):Topic                  = NULL     (0x03000000:NameValue):ContentType            = NULL     (0x03000000:NameValue):IdentitySourceType     = NULL     (0x03000000:NameValue):IdentitySourceToken    = NULL     (0x03000000:NameValue):IdentitySourcePassword = NULL     (0x03000000:NameValue):IdentitySourceIssuedBy = NULL     (0x03000000:NameValue):IdentityMappedType     = NULL     (0x03000000:NameValue):IdentityMappedToken    = NULL     (0x03000000:NameValue):IdentityMappedPassword = NULL     (0x03000000:NameValue):IdentityMappedIssuedBy = NULL   )   (0x01000000:Object):JSON       = ( ['json' : 0x7f68b81666a0]     (0x01000000:Object):Data = (       (0x01000000:Object):Employee = (         (0x03000000:NameValue):EMPNO    = '000200' (CHARACTER)         (0x03000000:NameValue):FIRSTNME = 'DAVID' (CHARACTER)         (0x03000000:NameValue):LASTNAME = 'BROWN' (CHARACTER)       )     )   ) ) ' from trace node 'employeeService.Trace'. 
Jun 28 20:38:08 bpmdemo IIB[77236]: IBM Integration Bus v9002 (IB9NODE.IIB9) [Thread 90570] (Msg 1/1) BIP3051E: Error message '( ['MQROOT' : 0x7f68b80d3e00]   (0x01000000:Name  ):Properties = ( ['MQPROPERTYPARSER' : 0x7f68b803c390]     (0x03000000:NameValue):MessageSet             = NULL     (0x03000000:NameValue):MessageType            = NULL     (0x03000000:NameValue):MessageFormat          = NULL     (0x03000000:NameValue):Encoding               = NULL     (0x03000000:NameValue):CodedCharSetId         = NULL     (0x03000000:NameValue):Transactional          = NULL     (0x03000000:NameValue):Persistence            = NULL     (0x03000000:NameValue):CreationTime           = NULL     (0x03000000:NameValue):ExpirationTime         = NULL     (0x03000000:NameValue):Priority               = NULL     (0x03000000:NameValue):ReplyIdentifier        = NULL     (0x03000000:NameValue):ReplyProtocol          = 'MQ' (CHARACTER)     (0x03000000:NameValue):Topic                  = NULL     (0x03000000:NameValue):ContentType            = NULL     (0x03000000:NameValue):IdentitySourceType     = NULL     (0x03000000:NameValue):IdentitySourceToken    = NULL     (0x03000000:NameValue):IdentitySourcePassword = NULL     (0x03000000:NameValue):IdentitySourceIssuedBy = NULL     (0x03000000:NameValue):IdentityMappedType     = NULL     (0x03000000:NameValue):IdentityMappedToken    = NULL     (0x03000000:NameValue):IdentityMappedPassword = NULL     (0x03000000:NameValue):IdentityMappedIssuedBy = NULL   )   (0x01000000:Object):JSON       = ( ['json' : 0x7f68b81666a0]     (0x01000000:Object):Data = (       (0x01000000:Object):Employee = (         (0x03000000:NameValue):EMPNO    = '000300' (CHARACTER)         (0x03000000:NameValue):FIRSTNME = 'PHILIP' (CHARACTER)         (0x03000000:NameValue):LASTNAME = 'SMITH' (CHARACTER)       )     )   ) ) ' from trace node 'employeeService.Trace'. 




Event Sequencing Service and IBM BPM Advanced 8.5.6.0

I was seeing this :-

[6/28/15 7:26:09:973 BST] 00000379 EsMeConnectio W com.ibm.wbiserver.sequencing.service.messaging.EsMeConnectionService run() CWRES0053W: The event sequencing service cannot connect to the messaging engine on the Service Component Architecture (SCA) system bus that hosts the destination sca/demo101/component/sendID.

whilst testing a BPEL component in IBM Integration Designer 8.5.6, against an IBM BPM Advanced 8.5.6.0 runtime.

This kinda came out of nowhere, but the key phrase event sequencing rang a bell.

I'd been fiddling around with the properties of my BPEL ( prior to realising that I'd not changed it from the default Long-Running Process to Microflow.

I'd specifically changed one of the Qualities of Service (QoS) parameters for one of my interfaces: -

specifically this one: -


In other words, I'd enabled Event Sequencing on the runtime, which was something that I did not specifically need.

Once I removed the Qualifier, and restarted the Process Server runtime, life was good again.

Saturday, 27 June 2015

IBM Business Process Manager and Desktop SSO via Kerberos and SPNEGO

Context

This comes on the back of a similar article describing how one can configure IBM Operational Decision Manager (ODM) 8.6 to work with Microsoft Active Directory 2008, and deliver desktop Single Sign-On for Windows users.

In this scenario, ODM has been "replaced" by IBM BPM Advanced 8.5.5.0 on IBM WebSphere Application Server (WAS) 8.5.5.2.

Again, the objective is to configure WAS to use Active Directory 2008 for identity management, authentication and authorisation ( access control ). In addition, Kerberos / SPNEGO will be configured in AD and in WAS, in order to allow Windows desktop users to automatically access BPM functionality without being challenged for credentials ( using their existing Windows log in ).

This article is broken into sections, and requires each section to be completed in order. In addition, it is assumed that AD has already been configured, and that the relevant users and groups have been created.

The following is intended to provide an example of the typical users and groups that one might require: -

WAS > AD Bind User

dn: CN=ldapbind,CN=Users,DC=ad2008,DC=uk,DC=ibm,DC=com

WAS Administration User

dn: CN=WebSphere Admin,CN=Users,DC=ad2008,DC=uk,DC=ibm,DC=com

WAS Administration Group

dn: CN=WebSphereAdmins,CN=Users,DC=ad2008,DC=uk,DC=ibm,DC=com
member: CN=WebSphere Admin,CN=Users,DC=ad2008,DC=uk,DC=ibm,DC=com

BPM Administration Group

dn: CN=BPMAdmins,CN=Users,DC=ad2008,DC=uk,DC=ibm,DC=com
member: CN=Administrator,CN=Users,DC=ad2008,DC=uk,DC=ibm,DC=com

BPM Authors Group

dn: CN=BPMAuthors,CN=Users,DC=ad2008,DC=uk,DC=ibm,DC=com
member: CN=Administrator,CN=Users,DC=ad2008,DC=uk,DC=ibm,DC=com

BPM Users Group

dn: CN=BPMUsers,CN=Users,DC=ad2008,DC=uk,DC=ibm,DC=com
member: CN=Administrator,CN=Users,DC=ad2008,DC=uk,DC=ibm,DC=com

Pre-Requisites

  • IBM BPM Advanced 8.5.5.0 installed on WebSphere Application Server Network Deployment 8.5.5.2, with IBM HTTP Server and IBM WebSphere Plugin.
  • IHS/Plugin federated into WAS cell
  • IHS configured for SSL
  • WAS Virtual Host for IHS SSL port added ( 8443 in this case )
  • IHS-hosted SSL certificate in WAS cell-default trust store

WAS Tasks

Start WSAdmin

/opt/IBM/WebSphere/AppServer/profiles/Dmgr01/bin/wsadmin.sh -lang jython

Create LDAP Repository

AdminTask.createIdMgrLDAPRepository('[-default true -id ad2008.uk.ibm.com -adapterClassName com.ibm.ws.wim.adapter.ldap.LdapAdapter -ldapServerType AD -sslConfiguration -certificateMapMode exactdn -supportChangeLog none -certificateFilter -loginProperties uid]')

Add AD Server

AdminTask.addIdMgrLDAPServer('[-id ad2008.uk.ibm.com -host ad2008.uk.ibm.com -bindDN CN=ldapbind,CN=Users,DC=ad2008,DC=uk,DC=ibm,DC=com -bindPassword Qk525mtp -referal ignore -sslEnabled false -ldapServerType AD -sslConfiguration -certificateMapMode exactdn -certificateFilter -authentication simple -port 389]')

Add Base Entry

AdminTask.addIdMgrRepositoryBaseEntry('[-id ad2008.uk.ibm.com -name CN=Users,DC=ad2008,DC=uk,DC=ibm,DC=com -nameInRepository CN=Users,DC=ad2008,DC=uk,DC=ibm,DC=com]')

Add Realm Base Entry

AdminTask.addIdMgrRealmBaseEntry('[-name defaultWIMFileBasedRealm -baseEntry CN=Users,DC=ad2008,DC=uk,DC=ibm,DC=com]')

Add User Object Classes and Search Base

AdminTask.updateIdMgrLDAPEntityType('[-id ad2008.uk.ibm.com -name PersonAccount -objectClasses user;organizationalPerson;person -searchBases CN=Users,DC=ad2008,DC=uk,DC=ibm,DC=com -searchFilter (ObjectCategory=User)]')

Add Group Object Class and Search Base

AdminTask.updateIdMgrLDAPEntityType('[-id ad2008.uk.ibm.com -name Group -objectClasses group -searchBases CN=Users,DC=ad2008,DC=uk,DC=ibm,DC=com -searchFilter (ObjectCategory=Group)]')

Enable Allow Operation If Repository Down

AdminTask.updateIdMgrRealm('[-name defaultWIMFileBasedRealm -allowOperationIfReposDown true]')

Add AD Group ( ) into Admin Roles

AdminTask.mapGroupsToAdminRole('[-roleName adminsecuritymanager -accessids [group:defaultWIMFileBasedRealm/CN=WebSphereAdmins,CN=Users,DC=ad2008,DC=uk,DC=ibm,DC=com ] -groupids [WebSphereAdmins@defaultWIMFileBasedRealm ]]')
AdminTask.mapGroupsToAdminRole('[-roleName administrator -accessids [group:defaultWIMFileBasedRealm/CN=WebSphereAdmins,CN=Users,DC=ad2008,DC=uk,DC=ibm,DC=com ] -groupids [WebSphereAdmins@defaultWIMFileBasedRealm ]]')
AdminTask.mapGroupsToAuditRole('[-roleName auditor -accessids [group:defaultWIMFileBasedRealm/CN=WebSphereAdmins,CN=Users,DC=ad2008,DC=uk,DC=ibm,DC=com ] -groupids [WebSphereAdmins@defaultWIMFileBasedRealm ]]')
AdminTask.mapGroupsToAdminRole('[-roleName configurator -accessids [group:defaultWIMFileBasedRealm/CN=WebSphereAdmins,CN=Users,DC=ad2008,DC=uk,DC=ibm,DC=com ] -groupids [WebSphereAdmins@defaultWIMFileBasedRealm ]]')
AdminTask.mapGroupsToAdminRole('[-roleName deployer -accessids [group:defaultWIMFileBasedRealm/CN=WebSphereAdmins,CN=Users,DC=ad2008,DC=uk,DC=ibm,DC=com ] -groupids [WebSphereAdmins@defaultWIMFileBasedRealm ]]')
AdminTask.mapGroupsToAdminRole('[-roleName iscadmins -accessids [group:defaultWIMFileBasedRealm/CN=WebSphereAdmins,CN=Users,DC=ad2008,DC=uk,DC=ibm,DC=com ] -groupids [WebSphereAdmins@defaultWIMFileBasedRealm ]]')
AdminTask.mapGroupsToAdminRole('[-roleName monitor -accessids [group:defaultWIMFileBasedRealm/CN=WebSphereAdmins,CN=Users,DC=ad2008,DC=uk,DC=ibm,DC=com ] -groupids [WebSphereAdmins@defaultWIMFileBasedRealm ]]')
AdminTask.mapGroupsToAdminRole('[-roleName operator -accessids [group:defaultWIMFileBasedRealm/CN=WebSphereAdmins,CN=Users,DC=ad2008,DC=uk,DC=ibm,DC=com ] -groupids [WebSphereAdmins@defaultWIMFileBasedRealm ]]')

Save and Sync

AdminConfig.save()
AdminNodeManagement.syncActiveNodes()
quit

AD Tasks

Create Service Principal Name

setspn -a HTTP/bpm855.uk.ibm.com ldapbind

Validate SPN

setspn -l ldapbind

Create Kerberos Keytab

ktpass -out bpm855.keytab -princ HTTP/ bpm855.uk.ibm.com@AD2008 -mapUser AD2008\ldapbind -mapOp set -pass Qk525mtp -ptype KRB5_NT_PRINCIPAL

WAS Tasks

Validate Kerberos Keytab

ktutil

ktutil: rkt bpm855.keytab
ktutil: list

slot KVNO Principal
---- ---- ---------------------------------------------------------------------
   1    3            HTTP/bpm855.uk.ibm.com@AD2008

ktutil: quit

/opt/IBM/WebSphere/AppServer/java/jre/bin/java com.ibm.security.krb5.internal.tools.Klist -e -k /home/wasadmin/bpm855.keytab

Key table: /home/wasadmin/bpm855.keytab
Number of entries: 1

[1] principal: HTTP/bpm855.uk.ibm.com@AD2008
KVNO: 3

Encryption type: RC4 with HMAC

Create Kerberos Configuration File

/opt/IBM/WebSphere/AppServer/profiles/Dmgr01/bin/wsadmin.sh -lang jython

AdminTask.createKrbConfigFile("[-krbPath /opt/IBM/WebSphere/AppServer/java/jre/lib/security/krb5.conf -realm UK.IBM.COM -kdcHost ad2008.uk.ibm.com.com -dns uk.ibm.com -keytabPath /home/wasadmin/bpm855.keytab -encryption des3-cbc-sha1]")

'/opt/IBM/WebSphere/AppServer/java/jre/lib/security/krb5.conf has been created.'

AdminConfig.save()
AdminNodeManagement.syncActiveNodes()
quit

Validate

cat /opt/IBM/WebSphere/AppServer/java/jre/lib/security/krb5.conf

[libdefaults]
   default_realm = UK.IBM.COM
   default_keytab_name = FILE:/home/
wasadmin/bpm855.keytab
   default_tkt_enctypes = des3-cbc-sha1
   default_tgs_enctypes = des3-cbc-sha1
   forwardable  = true
   renewable  = true
   noaddresses = true
   clockskew  = 300
[realms]
   UK.IBM.COM = {
     kdc = ad2008.uk.ibm.com.com:88
     default_domain = uk.ibm.com
   }
[domain_realm]

Add SPNEGO Configuration

/opt/IBM/WebSphere/AppServer/profiles/Dmgr01/bin/wsadmin.sh -lang jython
AdminTask.addSpnegoFilter('[-hostName bpm855.uk.ibm.com -krb5Realm AD2008 -filterCriteria -filterClass -spnegoNotSupportedPage -ntlmTokenReceivedPage -trimUserName true -enabledGssCredDelegate false]')
AdminTask.configureSpnego('[-enabled true -dynamicReload true -allowAppAuthMethodFallback true -krb5Config /opt/IBM/WebSphere/AppServer/java/jre/lib/security/krb5.conf -krb5Keytab /home/wasadmin/bpm855.keytab]')
AdminTask.setAdminActiveSecuritySettings('[-customProperties ["com.ibm.websphere.security.krb.canonical_host=true"] ]')
AdminConfig.save()
AdminNodeManagement.syncActiveNodes()
quit

Create Fallback Login Page - /opt/IBM/HTTPServer/htdocs/NoSpnegoRedirect.html

This page is used when (a) one uses a browser that is not configured for SPNEGO e.g. Chrome or Safari etc. and (b) when one chooses NOT to invoke SSO by adding ?noSPNEGO to the URL

<!DOCTYPE HTML PUBLIC "-//W3C/DTD HTML 4.0 Transitional//EN">
<META HTTP-EQUIV="Content-Type" CONTENT="text/html">
<!--
Notes:
- This file should be served from an unprotected website. Alternatively, it can be loaded from the WebSphere Application Server file system.
- Any imbedded graphics/javascript/css must be loaded from an unprotected website.
- This file is loaded after WebSphere Application Server is initialized. If changes to this file are necessary, restart WebSphere Application Server.
- This file is returned whenever the SPNEGO TAI receives an NTLM token for any application in the cell. In other words, this file is generic for all applications. However, by using the  document.location Javascipt,
- we can get the original URL, and redirect to that original URL with the "?noSPNEGO" text added - thus forcing the standard application userid/password challenge.
-->
<html>
<script language="javascript">
var origUrl=""+document.location;
   if (origUrl.indexOf("noSPNEGO")<0) {
if (origUrl.indexOf('?')>=0) origUrl+="&noSPNEGO";
else origUrl+="?noSPNEGO";
}
function redirTimer() {
self.setTimeout("self.location.href=origUrl;",0);
}
</script>

<META HTTP-EQUIV = "Pragma" CONTENT="no-cache">
<script language="javascript">
document.write("<title> Redirect to "+origUrl+ " </title>");
</script>
<head>
</head>
<body onLoad="redirTimer()"/>
</html>

Add Fallback Login Page to WAS

This also updates the SPNEGO filters to disable SSO between Process Center and Process Server - see SPNEGO authentication might cause communication issues between the Process Server and the Process Center for IBM Business Process Manager (BPM)

/opt/IBM/WebSphere/AppServer/profiles/Dmgr01/bin/wsadmin.sh -lang jython
AdminTask.modifySpnegoFilter('[-hostName bpm855.uk.ibm.com -krb5Realm AD2008 -filterCriteria request-url!=ProcessCenterInternal;request-url!=ProcessServerInternal;request-url^=ProcessCenter|ProcessAdmin|portal|ProcessPortal|teamworks|PerformanceAdmin;request-url!=noSPNEGO; -filterClass -spnegoNotSupportedPage https://bpm855.uk.ibm.com:8443/NoSpnegoRedirect.html -ntlmTokenReceivedPage https://bpm855.uk.ibm.com:8443/NoSpnegoRedirect.html -trimUserName true -enabledGssCredDelegate false]')
AdminConfig.save()
AdminNodeManagement.syncActiveNodes()
quit

Restart Deployment Environment

/opt/IBM/WebSphere/AppServer/bin/BPMConfig.sh -stop -profile Dmgr01 -de De1

/opt/IBM/WebSphere/AppServer/bin/BPMConfig.sh -start -profile Dmgr01 -de De1

Windows Desktop Tasks ( IE )

Enable Integrated Windows Authentication (IWA), if not already enabled: -

 

Add https://bpm855.uk.ibm.com to Local Intranet Trusted Zone: -




Restart IE and access Process Center URL: -


SSO should now work i.e. one should be logged into PC as the Windows user - Administrator: -


The same should work for Process Admin: -


and Process Portal: -


By default, however, SSO for Performance Admin: -


will not work, purely because the only user authorised to access Performance Admin is deAdmin.

This means that the login page will be presented, with a corresponding message in SystemOut.log ( for the SupportCluster ): -

[14/11/14 10:17:18:401 GMT] 00000143 WebCollaborat A   SECJ0129E: Authorization failed for user Administrator:defaultWIMFileBasedRealm while invoking GET on default_host:/PerformanceAdmin/console/Welcome.do, Authorization failed, Not granted any of the required roles: twuser

If needed, this could be mitigated by adding, for example, BPMAdmins into the twuser role for the Performance Admin application: -

/opt/IBM/WebSphere/AppServer/profiles/Dmgr01/bin/wsadmin.sh -lang jython
AdminApp.edit('IBM_BPM_PerformanceDW_SupCluster', '[ -MapRolesToUsers [[ twuser AppDeploymentOption.No AppDeploymentOption.No deAdmin BPMAdmins AppDeploymentOption.No " " group:defaultWIMFileBasedRealm/CN=BPMAdmins,CN=Users,DC=ad2008,DC=uk,DC=ibm,DC=com ]]]' )
AdminConfig.save()
AdminNodeManagement.syncActiveNodes()
quit

However, this requires insight from the business, as the security requirements for the Performance Admin console may differ to those for Process Center, Process Portal, Process Admin etc.

Optional and Additional Steps

BPM internal groups can be updated using Process Admin in order to enforce specific roles when using SSO: -

Internal BPM Group AD Group

tw_authors BPMAuthors
tw_admins BPMAdmins
tw_allusers BPMUsers

Success in Pictures

Process Center


Process Portal


Process Admin


Performance Admin


References






Thursday, 25 June 2015

WAS and Kerberos and SPNEGO - it's been a while - but it's all right now, in fact it it's a gas

It's been a while since first I worked with WAS, Kerberos and SPNEGO - 2011 was SUCH a long time ago.

We're configuring desktop SSO between WAS ND 8.5.5.0 ( specifically for BPM Advanced 8.5.5.0 but that's not important right now ) and MS Active Directory 2008.

I'd previously documented the required steps to create the Service Principal Name (SPN) and Kerberos key tab in AD, and I was confident that there were no issues there.

However, when we started up the WAS JVM ( we were specifically looking at the Deployment Manager ), we saw the following in SystemOut.log: -

[25/06/15 14:05:19:001 BST] 0000015a ServerCredent E com.ibm.ws.security.spnego.ServerCredential initialize CWSPN0014E: An exception occurred during Kerberos initialization. Failure: org.ietf.jgss.GSSException, major code: 13, minor code: 0 major string: Invalid credentials minor string: Cannot get credential from JAAS Subject for principal: HTTP/bpm101@2008ADMINLDAPS.

[25/06/15 14:05:19:008 BST] 0000015a ServerCredent E com.ibm.ws.security.spnego.ServerCredentialsFactory initializeServer CWSPN0015E: Unable to create a GSSCredential for: HTTP/bpm101@2008ADMINLDAPS

[25/06/15 14:05:19:008 BST] 0000015a ServerCredent E com.ibm.ws.security.spnego.ServerCredentialsFactory initializeServerCreds CWSPN0017E: Unable to create GSSCredentials for any of the hosts specified in the configuration properties.

[25/06/15 14:05:19:048 BST] 0000015a TrustAssociat E com.ibm.ws.security.spnego.TrustAssociationInterceptorImpl initialize CWSPN0009E: SPNEGO Trust Association Interceptor configuration is not valid. Failure condition: initializeServerCreds: no credentials successfully created.. If you are not using the SPNEGO TAI, you can ignore this message.


and, in the FFDC logs: -

org.ietf.jgss.GSSException, major code: 13, minor code: 0 major string: Invalid credentials
minor string: Cannot get credential from JAAS Subject for principal: HTTP/bpm101@2008ADMINLDAPS
 at com.ibm.security.jgss.i18n.I18NException.throwGSSException(I18NException.java:35)

[25/06/15 14:05:19:005 BST]     FFDC Exception:org.ietf.jgss.GSSException SourceId:com.ibm.ws.security.spnego.ServerCredentialsFactory.initializeServer ProbeId:68 org.ietf.jgss.GSSException, major code: 13, minor code: 0 major string: Invalid credentials
minor string: Cannot get credential from JAAS Subject for principal: HTTP/bpm101@2008ADMINLDAPS

[25/06/15 14:05:19:023 BST]     FFDC Exception:com.ibm.ws.security.spnego.TAIConfigurationException SourceId:com.ibm.ws.security.spnego.TrustAssociationInterceptorImpl.initialize ProbeId:332 Reporter:com.ibm.ws.security.spnego.TrustAssociationInterceptorImpl@7238f7dd

com.ibm.ws.security.spnego.TAIConfigurationException: initializeServerCreds: no credentials successfully created.


We started by checking the WAS configuration and then moved on to the Kerberos key tab: -

/opt/ibm/WebSphere/AppServer/java/jre/bin/java com.ibm.security.krb5.internal.tools.Klist -e -k /home/wasadmin/keytabs/bpm.keytab

which returns: -

Key table: /home/wasadmin/keytabs/bpm.keytab
Number of entries: 1

1] principal: HTTP/bpm101.uk.ibm.com@2008ADMINLDAPS
        KVNO: 4

       Encryption type: RC4 with HMAC

We also checked the WAS Kerberos configuration file, krb5.conf.

[libdefaults]
        default_realm = uk.ibm.com
        default_keytab_name = FILE:/home/wasadmin/keytabs/bpm.keytab
        default_tkt_enctypes = des3-cbc-sha1
        default_tgs_enctypes = des3-cbc-sha1
        forwardable  = true
        renewable  = true
        noaddresses = true
        clockskew  = 300

[realms]
        uk.ibm.com = {
                kdc = 2008ADMINLDAPS.uk.ibm.com:88
                default_domain = uk.ibm.com
        }

[domain_realm]
        .uk.ibm.com = uk.ibm.com

We then started to look at the hostname of the server, as compared with what DNS shows.

Looking back at an older blog post ( from June 2014 ): -


I compiled and tested a Java class: -

import java.net.InetAddress;
import java.net.UnknownHostException;

public class hostStuff
{
public static void main(String[] args)
{

try
{
InetAddress address = InetAddress.getLocalHost();
System.out.println("My IP address ( via InetAddress.getLocalHost() ) is " + address.toString());
System.out.println("My hostname ( via InetAddress.getHostName() ) is " + address.getHostName());
System.out.println("My hostname ( via InetAddress.getCanonicalHostname() ) is  " + address.getCanonicalHostName());
}
catch (UnknownHostException e)
{
       System.out.println("I'm sorry. I don't know my own name.");
}
}
}

which showed that, whilst we were seeing the correct hostname, we were ONLY seeing the short name, rather than the fully-qualified service name.

In other words, we were seeing bpm101 rather than bpm101.uk.ibm.com.

When we checked further, that's what /etc/hosts was configured to use.

We asked the AIX sysadmin to change this - he commented out the entry in the hosts file - and then restarted WAS.

Immediately, SPNEGO burst into life.

Bottom line, it doesn't matter how good the SPNEGO and WAS configuration is, it is CRUCIAL to ensure that the hostname that WAS can see, via the various Java methods, matches what's been configured for the Service Principal Name (SPN).

Life is, as they say, a learning curve :-)



Oh Dear Busy Crying - Where's my ODBC data source ?

This is definitely a "Slap myself" moment.

Running my IBM Integration Bus (IIB) flow, the one that goes to DB2 via ODBC, produced this in /var/log/user.log: -

Jun 24 18:39:30 bpmdemo IIB[4439]: IBM Integration Bus v9002 (IB9NODE.IIB9) [Thread 4553] (Msg 1/4) BIP2628E: Exception condition detected on input node 'customerService.Input'. 
Jun 24 18:39:30 bpmdemo IIB[4439]: IBM Integration Bus v9002 (IB9NODE.IIB9) [Thread 4553] (Msg 2/4) BIP2230E: Error detected whilst processing a message in node 'customerService.Compute'. 
Jun 24 18:39:30 bpmdemo IIB[4439]: IBM Integration Bus v9002 (IB9NODE.IIB9) [Thread 4553] (Msg 3/4) BIP2393E: Database error: ODBC return code '-1' from data source 'SAMPLE' using ODBC driver manager ''. 
Jun 24 18:39:30 bpmdemo IIB[4439]: IBM Integration Bus v9002 (IB9NODE.IIB9) [Thread 4553] (Msg 4/4) BIP2347E: Database error: SQL State 'IM002'; Native Error Code '0'; Error Text 'System information file not found. Please check the ODBCINI and the ODBCSYSINI environment variables are correctly set.' 
Jun 24 18:39:31 bpmdemo IIB[4439]: IBM Integration Bus v9002 (IB9NODE.IIB9) [Thread 4553] (Msg 1/1) BIP2648E: Message backed out to a queue; node 'customerService.Input'. 

The reason was obvious to me, even though I'd forgotten to do it :-)

When I start IIB, I need to ensure that a pair of variables are set: -

export ODBCINI=/etc/odbc.ini 
export ODBCSYSINI=/etc


or the Bus cannot "see" the ODBC data source.

Short term, I scripted the startup of IIB thusly: -

startIIB.sh

#! /bin/bash
export ODBCINI=/etc/odbc.ini 
export ODBCSYSINI=/etc
mqsistart IB9NODE


but I could've easily set them in the owning user's ( wmbadmin ) startup script - .bashrc or .profile.

Which is nice :-)