Monday, 31 July 2017

Oracle Service Names - I *WILL* get this right - EVENTUALLY

I saw this: -

Bootstraping data into cluster AppCluster and logging into /opt/ibm/WebSphereProfiles/Dmgr01/logs/bootstrapProcesServerData.AppCluster.log

WASX7357I: By request, this scripting client is not connected to any server process. Certain configuration and application operations will be available in local mode.
java.sql.SQLRecoverableException: Listener refused the connection with the following error:
ORA-12514, TNS:listener does not currently know of service requested in connect descriptor

...
Caused by: oracle.net.ns.NetException: Listener refused the connection with the following error:
ORA-12514, TNS:listener does not currently know of service requested in connect descriptor

...
 
whilst running the bootstrap process for an IBM BPM environment: -

/opt/ibm/WebSphereProfiles/Dmgr01/bin/bootstrapProcessServerData.sh -clusterName AppCluster

Again, this is because I keep getting confused between the Oracle SID and the service name.

This helped: -


sqlplus / as sysdba

SQL*Plus: Release 12.2.0.1.0 Production on Mon Jul 31 15:18:15 2017

Copyright (c) 1982, 2016, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Standard Edition Release 12.2.0.1.0 - 64bit Production

SQL> select sys_context('userenv','db_name') from dual;

SYS_CONTEXT('USERENV','DB_NAME')
--------------------------------------------------------------------------------
orcl

SQL> select ora_database_name from dual;

ORA_DATABASE_NAME
--------------------------------------------------------------------------------
ORCL.UK.IBM.COM

SQL> exit
Disconnected from Oracle Database 12c Standard Edition Release 12.2.0.1.0 - 64bit Production

Once I realised which was which, I updated my JDBC datasources to reflect the correct Service Name ( oracle.uk.ibm.com ) and all is now well: -

/opt/ibm/WebSphereProfiles/Dmgr01/bin/bootstrapProcessServerData.sh -clusterName AppCluster

Bootstraping data into cluster AppCluster and logging into /opt/ibm/WebSphereProfiles/Dmgr01/logs/bootstrapProcesServerData.AppCluster.log

WASX7357I: By request, this scripting client is not connected to any server process. Certain configuration and application operations will be available in local mode.
'BootstrapProcessServerData admin command completed successfully.....'


For the record, this WAS the URL that I had in my datasource: -

jdbc:oracle:thin:@//bpm857.uk.ibm.com:1521/orcl

and this is what I now have: -


Solved, WebSphere eats Linux - or, Linux reboots when WAS JVM starts up

So I've been working an interesting "bug" over the past few days.

The long story short is that we had a VM running Red Hat Enterprise Linux 6.6 ( with kernel 2.6.32-696.3.1.el6.x86_64 ), hosting WebSphere Application Server 8.5.5.8 and BPM Standard 8.5.7.0.

BPM is installed a single cluster, with all of the workload running in a single JVM - in this instance.

Whilst I could happily start the Deployment Manager and Node Agent, when I started the actual BPM JVM, after a minute or two, the box would reboot.

This was 100% reproducible.

I spent a happy day last week debugging this, looking at class paths, dependent JARs, auto starting EAR files etc. but to no avail.

I even wondered whether the fact that we were running slightly older versions of WAS and BPM was pertinent, even though I've used them a million times before.

Interestingly, nothing showed up in dmesg or /var/log/messages or /var/log/kernel.log and there was no obvious kernel panic messages therein.

Thankfully, with the help of a VMware SME, we did find this : -

tail -f /var/crash/127.0.0.1-2017-07-31-10:43:30

...
<2>kernel BUG at drivers/net/vmxnet3/vmxnet3_drv.c:1412!
<4>invalid opcode: 0000 [#1] SMP
<4>last sysfs file: /sys/devices/system/cpu/online


which tied up with this: -


which says, in part : -

<snip>
This issue affects all virtual machines running on ESXi 6.5 host (with virtual hardware version 13), the guest will freeze randomly (sometimes several minutes after power on, and sometimes freezes several hours from boot).

I got this kernel panic log several times, possibly this issue was caused by VMXNET3.

All kernel newer than 4.8.x are affected with this issue, if I downgrade the kernel version back to 4.4.x, the VMs will work like a charm.

(Guest OS is CentOS 7.3 with kernel-ml)

And this issue doesn't happen while virtual hardware version 11 with ESXi 6.5, only happen on virtual hardware version 13 + ESXi 6.5.
</snip>

My friendly VMware SME spotted this, and suggested that we switch the virtual Network Interface Card (vNIC) to use a different driver - namely e1000.

We did this and …. voila :-)

The BPM runtime starts happily and all is good.

This appears to be related to the specific version of VMware ESX ( aka vSphere ) and the fact that the Linux VM is newly created, using the most recent virtual hardware version.

So that's all good then ….

Wednesday, 26 July 2017

WebSphere Application Server - Scripting Security

I'm creating a set of scripts to automate a number of common tasks, including setting up Web SSO: -

/mnt/Scripts/enableSSO.jy 


AdminTask.configureSingleSignon(['-enable', 'true', '-requiresSSL', 'true', '-domainName', 'uk.ibm.com'])

# Save and Sync

AdminConfig.save()

AdminNodeManagement.syncActiveNodes()
 
/mnt/Scripts/exportLTPAkeys.jy

AdminTask.exportLTPAKeys('[-ltpaKeyFile file:/mnt/Misc/bpm855.uk.ibm.com.ltpa -password passw0rd ]')

/mnt/Scripts/importLTPAkeys.jy


AdminTask.configureSingleSignon(['-enable', 'true', '-requiresSSL', 'true', '-domainName', 'uk.ibm.com'])

# Save and Sync

AdminConfig.save()

AdminNodeManagement.syncActiveNodes()

With thanks to this: -



WebSphere Liberty Profile - Variables

As per my previous posts, I'm setting up an IBM BPM Process Federation Server (PFS), which runs on WebSphere Liberty Profile (WLP).

I'm now going back through the setup from scratch, to see (a) what I learned and (b) whether I can repeat my success :-)

Looking at the WLP configuration file - server.xml - I'd noticed reference to a pair of variables: -

    <keyStore id="defaultKeyStore" location="${server.output.dir}/resources/security/key.jks" password="password" />
    <keyStore id="defaultTrustStore" location="${server.config.dir}/resources/security/key.jks" password="password"/>

and was trying to work out what, if anything, the difference is, given that they actually point to a the same darn file.

This helped: -


WLP_OUTPUT_DIR

This environment variable can be used to specify an alternative location for server generated output such as logs, the workarea directory, and generated files. Files in the logs directory can include console.log, messages.log, and any generated FFDC files. Generated files can include server dumps that are created with the server dump or server javadump command. This variable must be an absolute path. If this environment variable is specified, ${server.output.dir} is set to the equivalent of WLP_OUTPUT_DIR/serverName. If this environment variable is not specified, ${server.output.dir} is the same as ${server.config.dir}.

So, because I have not actively specified WLP_OUTPUT_DIR  the value of server.output.dir is the same as server.config.dir.

For the record, I can also dump out the value of the overall WLP environment: -

/opt/ibm/PFS/v8.5/bin/server status

CWWKE0005E: The runtime environment could not be launched.
CWWKE0010E: The required server.xml file must exist and be readable. Path: /opt/ibm/PFS/v8.5/usr/servers/defaultServer/server.xml Reason: file not found
                 Java home:  /opt/ibm/PFS/v8.5/java/java_1.8_64/jre
              Install root:  /opt/ibm/PFS/v8.5/
          System libraries:  /opt/ibm/PFS/v8.5/lib/
                 User root:  /opt/ibm/PFS/v8.5/usr/
                    Config:  /opt/ibm/PFS/v8.5/usr/servers/defaultServer/
                    Output:  /opt/ibm/PFS/v8.5/usr/servers/defaultServer/

As I did not specify a server name, it dumps out the overall WLP configuration, which is helpful

Moving an IBM BPM 8.5.7 environment, dealing with the Service Integration Bus (SIBus ) tables

I'm working through this process: -


specifically testing the process of "lifting and shifting" an IBM BPM Deployment Environment from one VM to another.

This is, in part, to help a client move a running environment from Red Hat Enterprise Linux v5 to v7.

So I've got a working environment on a VM, and I've exported the BPM Deployment Environment thus: -

/opt/ibm/WebSphere/AppServer/bin/BPMConfig.sh -export -de De1 -profile Dmgr01 -outputDir /tmp/De1

deleted the WAS profiles: -

/opt/ibm/WebSphere/AppServer/bin/manageprofiles.sh -deleteAll

removed the WAS profile root: -

rm -Rf WebSphereProfiles/

and then recreated the Deployment Environment using the exported configuration: -

/opt/ibm/WebSphere/AppServer/bin/BPMConfig.sh -create -de  /tmp/De1/De1.properties 

I then need to do this: -


Manually drop the existing messaging engine tables in the messaging database of your new deployment environment before you start the deployment environment.

The messaging engine table names use the SIB prefix.
...

even though the underlying Oracle database isn't moving anywhere.

So, I'm on the Oracle box, and have started the SQLPlus client: -

sqlplus / as sysdba

Now I need to find the tables: -

SELECT TABLE_NAME FROM ALL_tables WHERE TABLE_NAME LIKE '%SIB%' AND OWNER = 'CMNUSER';

which gives me this: -

TABLE_NAME
--------------------------------------------------------------------------------
DATA_VISIBILITY_TEMPLATE_T
SIB000
SIB001
SIB002
SIBCLASSMAP
SIBKEYS
SIBLISTING
SIBOWNER
SIBOWNERO
SIBXACTS


10 rows selected.

I've highlighted the ones in which I'm interested: -

SIB000
SIB001
SIB002
SIBCLASSMAP
SIBKEYS
SIBLISTING
SIBOWNER
SIBOWNERO
SIBXACTS

so I merely need to drop them: -

vi dropSIBtables.sql

DROP TABLE CMNUSER.SIB000;
DROP TABLE CMNUSER.SIB001;
DROP TABLE CMNUSER.SIB002;
DROP TABLE CMNUSER.SIBCLASSMAP;
DROP TABLE CMNUSER.SIBKEYS;
DROP TABLE CMNUSER.SIBLISTING;
DROP TABLE CMNUSER.SIBOWNER;
DROP TABLE CMNUSER.SIBOWNERO;
DROP TABLE CMNUSER.SIBXACTS;

exit | sqlplus / as sysdba @dropSIBtables.sql

SQL*Plus: Release 12.2.0.1.0 Production on Wed Jul 26 11:25:45 2017

Copyright (c) 1982, 2016, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production


Table dropped.


Table dropped.


Table dropped.


Table dropped.


Table dropped.


Table dropped.


Table dropped.


Table dropped.


Table dropped.

SQL> Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production


Finally, I need to recreate the SIB tables, using the createSchema_Messaging.sql script that i used when I first built the environment: -

exit | sqlplus / as sysdba @createSchema_Messaging.sql

SQL*Plus: Release 12.2.0.1.0 Production on Wed Jul 26 11:31:36 2017

Copyright (c) 1982, 2016, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production


Table created.


Table created.


Table created.


Table created.


Table created.


Index created.


Table created.


Index created.


Table created.


Index created.


Table created.


Table created.

SQL> Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production


and then restart my Deployment Environment: -

/opt/ibm/WebSphere/AppServer/bin/BPMConfig.sh -start -profile Dmgr01 -de De1

and on to testing ….

Hmm, Box Sync not playing nice - "Unable to Connect to Box"

So I started seeing this: -

Unable to Connect to Box

Box Sync is unable to connect to Box.

Verify that you are connected to the Internet and restart Box Sync.

If you continue to see this message, contact your administrator to verify if it is a network issue.

If it is not a network issue, visit the Box Support website to submit a ticket.

yesterday, on my Mac.

No matter how many times I tried stopping/starting the Box Sync app, it flatly refused to connect.

This whilst I was working away on Box via my browser, and happily using the internet for mail, browsing, Slack, Twitter, iMessage etc.

So, this morning, I took Roy's advice ( if you don't know The IT Crowd, you really should ) and "turned it off and on again" - by which I mean the Mac itself.

A reboot didn't seem to sort it, so a full power-down and restart was the order of the day.

When I then tried to start the Box Sync client I saw this: -

There was a login error. Please make sure your network is connected and try logging in again. If you still experience issues, please restart Box Sync. Reason Code: (8)

so I again restart the Box Sync client itself ….

And, quelle surprise, I was able to log in and get syncing … which is nice :-)

Tuesday, 25 July 2017

IBM BPM and the Process Federation Server - A Voyage of Discovery #3

So, following these two posts: -



once I'd got PFS set up and (mostly) working, I did some testing.

However, I kept seeing errors such as this: -


when I logged into the BPM 8.5.7 Process Portal ( if you remember, this is the new go-to Portal for everything, including the 8.5.5 instances ) via this URL: -


Talking to some of my genius colleagues, I realised that I could debug this better using a JavaScript console such as that provided by Google Chrome.

This is what I saw: -


A spot of Googling for net::ERR_INSECURE_RESPONSE made me realise the error of my ways.

I'm using self-signed certificates for the IHS instances fronting BPM 8.5.5 and 8.5.7. In addition, I'm also using a self-signed certificate for PFS which, although not using IHS, still has a web UI sitting on port 9443: -


When I first access Process Portal or PFS in Chrome, I see this: -


In other words, whilst I'd told Chrome to "trust" BPM 8.5.7 I had not told it to similarly trust BPM 8.5.7.

Once I accessed PFS: -


and reloaded the original 8.5.7 Process Portal page, suddenly my instances started to appear …

I did see one subsequent issue: -

XMLHttpRequest cannot load https://bpm855.uk.ibm.com:8443/rest/bpm/wle/v1/task/56?federationMode=true&parts=actions. The 'Access-Control-Allow-Origin' header has a value 'https://bpm855.uk.ibm.com:9443' that is not equal to the supplied origin. Origin 'https://bpm857.uk.ibm.com:8443' is therefore not allowed access.

It took me a while to work this out, but then I realised that it's telling me that the IHS fronting BPM 8.5.5 is being asked to allow the HTTP header to be modified to read https://bpm857.uk.ibm.com:8443 which doesn't match up with the mod_headers directive of: -

Header set Access-Control-Allow-Origin "https://bpm855.uk.ibm.com:8443"

which makes no sense at all, given that it's on the BPM 8.5.5 box itself :-)

Once I changed my IHS configuration: -

LoadModule headers_module modules/mod_headers.so
Header set Access-Control-Allow-Origin "https://bpm857.uk.ibm.com:8443"
Header set Access-Control-Allow-Credentials "true"
Header set Access-Control-Allow-Headers "DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type"
Header set Access-Control-Allow-Methods "GET,POST,PUT,DELETE,OPTIONS"


all was well.

I'm still on the learning curve, especially with regard to the specifics of SSL certificate exchange, so expect more posts ….

IBM BPM and the Process Federation Server - A Voyage of Discovery #2

So, following on from my earlier post: -


it took me a while to realise that PFS is NOT a UI in its own right; it merely aggregates processes from other BPMs into one single Process Portal.

However, YOU need to provide that Process Portal :-)

So, in my limited test rig here, I have four VMs: -

BPM 8.5.5 This is my source environment, from which I am migrating, which has running instances which I need to drain down over time
BPM 8.5.7 This is my target environment, to which I am NOW going to send my users, upon which they will run new instances AND access their old instances
PFS This is running the ElasticSearch engine etc.
DB2 With three instances; db2inst1 (BPM 8.5.5), db2inst2 (BPM 8.5.7) and db2inst3 (PFS)

Interestingly, PFS is a WebSphere Liberty-based runtime, which is nice.

Each of the BPM servers also has a local instance of IBM HTTP Server, which helps me simplify things.

Five key things I've learned: -

  • BPM 8.5.7 becomes the target UI; effectively PFS is "feeding" tasks from the BPM 8.5.5 box AND the BPM 8.5.7 box into the same Process Portal
  • The two BPMs *and* PFS need to either share a common user-registry e.g. LDAP, or the users need to exist ( with identical IDs and passwords ) in all three systems
  • SSL between the three components is important; this requires the exchange of self-signed certificates unless we're using CA-signed ( which I'm not in this particular PoC )
  • LTPA keys need to be exchanged between all three components; I generated a LTPA key on the BPM 8.5.5 box, and then shared it with the other two
  • Cross-Origin Resource Sharing (CORS) and XML HTTP Request (XHR) are now things I never knew I need to know - this is all related to the fact that the BPM 8.5.7 Process Portal is providing me with a list of tasks from BOTH BPMs, and has to generate URLs that relate to both
  • Similarly, Apache mod_headers and the Access-Control-Allow-Origin header are also my new BFFs

This is fun ….

IBM BPM and the Process Federation Server - A Voyage of Discovery #1

So I've spent the past week getting to grips with Process Federation Server (PFS), and I think I'm pretty close now.

In essence, PFS is a capability that's been around for a few years, and allows a business to provide an aggregation layer across multiple different Process Server environments.

Whilst this is often pertinent in the context of a migration or upgrade, perhaps where an older version of BPM is being maintained, to allow existing process instances to drain down, whilst new work is being initiated on a new environment, there's another use case for PFS.

For example, where a business wants to segregate workloads onto different environments, perhaps with a Bronze/Silver/Gold SLA-driven approach, then PFS can provide the "glue" between the three environments, allowing a user to "find their process".

PFS includes an ElasticSearch engine which maintains an index of inflight instances of BPMN ( BPD ) and SCA/BPEL processes.

Here's a picture: -




Install this component to create a federated process environment that provides business users with a single point of access to their task list and launch list, regardless of the type of process that they are working on and the IBM BPM back-end system on which the process artifacts are stored.

Consider installing Process Federation Server in the following situations:

• Your process applications are deployed on different IBM BPM deployment environments. Your users must log on to several different instances of Process Portal to get their work done.
• Your users work with both BPD-related and BPEL-related processes and tasks.
• Your IBM BPM environment contains multiple versions of IBM BPM that each runs a different version of the same process application. Your users need to work with the older versions of the process applications while process instances are still running, which means that they need to log on to the Process Portal or Heritage Process Portal application that runs on the associated version of IBM BPM.
Process Federation Server aggregates both BPD and BPEL-related tasks from all the IBM BPM back-end systems, including systems that run previous versions of IBM BPM that are supported by Process Federation Server. You can take advantage of this support to gradually phase out process applications that are still running on previous versions, while new versions of the process applications run on the latest release of IBM BPM

So this is the first of many posts about PFS …..

8.5.5.12: WebSphere Application Server V8.5.5 Fix Pack 12

I missed this on Friday: -



Time to get downloading ….

Monday, 24 July 2017

java.sql.SQLException: IO Error: The Network Adapter could not establish the connection DSRA0010E: SQL State = 08006, Error Code = 17,002

I saw this: -


The test connection operation failed for data source BPM Business Process Choreographer data source on server nodeagent at node Node1 with the following exception: java.sql.SQLException: IO Error: The Network Adapter could not establish the connection DSRA0010E: SQL State = 08006, Error Code = 17,002. View JVM logs for further details.

whilst testing JDBC data sources configured to connect to a newly-minted Oracle 12c database.

I checked the Oracle box to ensure that the listener was running: -

netstat -aon | grep LISTEN

tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      off (0.00/0/0)
tcp        0      0 127.0.0.1:1521          0.0.0.0:*               LISTEN      off (0.00/0/0)
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      off (0.00/0/0)
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      off (0.00/0/0)
tcp        0      0 127.0.0.1:6010          0.0.0.0:*               LISTEN      off (0.00/0/0)
tcp6       0      0 :::36620                :::*                    LISTEN      off (0.00/0/0)


but I wasn't able to connect using telnet from either the BPM 8.5.7 VM or, more interestingly, the Oracle box itself: -

telnet oracle 1521

Trying 192.168.153.131...
telnet: connect to address 192.168.153.131: Connection refused

so it felt more like networking or, to be more accurate, nOtworking :-)

I again checked the listener: -

lsnrctl status

LSNRCTL for Linux: Version 12.2.0.1.0 - Production on 24-JUL-2017 15:21:04

Copyright (c) 1991, 2016, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=localhost)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 12.2.0.1.0 - Production
Start Date                24-JUL-2017 13:26:23
Uptime                    0 days 1 hr. 54 min. 41 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /home/oracle/app/oracle/product/12.2.0/dbhome_1/network/admin/listener.ora
Listener Log File         /home/oracle/app/oracle/diag/tnslsnr/oracle/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=127.0.0.1)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))
Services Summary...
Service "orcl.uk.ibm.com" has 1 instance(s).
  Instance "orcl", status READY, has 1 handler(s) for this service...
Service "orclXDB.uk.ibm.com" has 1 instance(s).
  Instance "orcl", status READY, has 1 handler(s) for this service...
The command completed successfully


I then started to dig into the OS more: -

sestatus 

SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          disabled
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      28


service firewalld status

Redirecting to /bin/systemctl status  firewalld.service
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:firewalld(1)

Jul 24 12:31:09 oracle.uk.ibm.com systemd[1]: Starting firewalld - dynamic firewall daemon...
Jul 24 12:31:10 oracle.uk.ibm.com systemd[1]: Started firewalld - dynamic firewall daemon.
Jul 24 11:33:20 oracle.uk.ibm.com systemd[1]: Stopping firewalld - dynamic firewall daemon...
Jul 24 11:33:20 oracle.uk.ibm.com systemd[1]: Stopped firewalld - dynamic firewall daemon.


Wondering whether selinux was the problem, I changed the configuration: -

vi /etc/selinux/config

changing from: -

SELINUX=enforcing

to: -

SELINUX=disabled

and rebooted, but to no avail.

I then tried this: 

telnet localhost 1521

from the Oracle box itself.

Which worked ….

So I revisited the lsnrctl status again 


Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=127.0.0.1)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))

I started wondering whether the listener was "bound" to the 127.0.0.1 address ….

So I reconfigured it …

vi /home/oracle/app/oracle/product/12.2.0/dbhome_1/network/admin/listener.ora

changing from: -

# listener.ora Network Configuration File: /home/oracle/app/oracle/product/12.2.0/dbhome_1/network/admin/listener.ora
# Generated by Oracle configuration tools.

LISTENER =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = TCP)(HOST = localhost)(PORT = 1521))
      (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1521))
    )
  )

to: -

# listener.ora Network Configuration File: /home/oracle/app/oracle/product/12.2.0/dbhome_1/network/admin/listener.ora
# Generated by Oracle configuration tools.

LISTENER =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = TCP)(HOST = oracle.uk.ibm.com)(PORT = 1521))
      (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1521))
    )
  )

and restarted it: -

lsnrctl stop

LSNRCTL for Linux: Version 12.2.0.1.0 - Production on 24-JUL-2017 15:41:45

Copyright (c) 1991, 2016, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=oracle.uk.ibm.com)(PORT=1521)))
TNS-12541: TNS:no listener
 TNS-12560: TNS:protocol adapter error
  TNS-00511: No listener
   Linux Error: 111: Connection refused
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC1521)))
The command completed successfully

lsnrctl start

LSNRCTL for Linux: Version 12.2.0.1.0 - Production on 24-JUL-2017 15:41:56

Copyright (c) 1991, 2016, Oracle.  All rights reserved.

Starting /home/oracle/app/oracle/product/12.2.0/dbhome_1/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 12.2.0.1.0 - Production
System parameter file is /home/oracle/app/oracle/product/12.2.0/dbhome_1/network/admin/listener.ora
Log messages written to /home/oracle/app/oracle/diag/tnslsnr/oracle/listener/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=oracle.uk.ibm.com)(PORT=1521)))
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=oracle.uk.ibm.com)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 12.2.0.1.0 - Production
Start Date                24-JUL-2017 15:41:56
Uptime                    0 days 0 hr. 0 min. 0 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /home/oracle/app/oracle/product/12.2.0/dbhome_1/network/admin/listener.ora
Listener Log File         /home/oracle/app/oracle/diag/tnslsnr/oracle/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=oracle.uk.ibm.com)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))
The listener supports no services
The command completed successfully


and now telnet works: -

telnet `hostname` 1521

Trying 192.168.153.131...
Connected to oracle.uk.ibm.com.
Escape character is '^]'.
^]quit

telnet>
quit
Connection closed.

More importantly …

Further adventures with Oracle database - ORA-01078 and LRM-00109

Having installed Oracle 12c ( 12.2.0.1.0 ) today, I was somewhat concerned when I tried to start my database: -

sqlplus / as sysdba

SQL*Plus: Release 12.2.0.1.0 Production on Mon Jul 24 14:04:56 2017

Copyright (c) 1982, 2016, Oracle.  All rights reserved.

Connected to an idle instance.

SQL>
startup
ORA-01078: failure in processing system parameters
LRM-00109: could not open parameter file '/home/oracle/app/oracle/product/12.2.0/dbhome_1/dbs/initorcl.ora'
SQL> quit
Disconnected

I checked the offending directory: -

ls $ORACLE_HOME/dbs

init.ora

which made me think …..

So I re-ran the dbca command to see what was going on.

Guess what ?

I didn't have a database; I'd obviously installed the product BUT not created a database.

Once I created one, all is well: -

ls $ORACLE_HOME/dbs

hc_orcl.dat  init.ora  lkORCL  orapworcl  spfileorcl.ora

sqlplus / as sysdba

SQL*Plus: Release 12.2.0.1.0 Production on Mon Jul 24 14:53:27 2017

Copyright (c) 1982, 2016, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL> exit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production


Sunday, 16 July 2017

ilog.rules.teamserver.model.IlrConnectException Caused by: java.lang.NullPointerException

As one of my many spinning threads, I've been tinkering with the migration of an IBM Operational Decision Manager (ODM) environment from version 8.0 ( circa 2012 ) to version 8.9 ( circa 2017 ).

In part, this involves running some SQL scripts to migrate the old data to the new environment.

This is especially relevant for the Decision Center, which is the Rules authoring environment, aka the Source Code Management (SCM) system-of-record.

The Decision Center, also known as the Team Server ( reflecting the product's heritage as iLog JRules ) or Rule Team Server or RTS, is based upon a database.

Therefore, the first step is to BACKUP the existing database ( RTSDB, in my case ) from the old/source system.

One benefit of the migration is that it's effectively side-by-side, meaning that you leave the old source system alone, and build a new, parallel environment into which the data ( Rules etc. ) is restored.

In my case, I'm using IBM DB2, but other databases are available.

The process is documented here: -


and here: -


and involves running a pair of Ant scripts against the existing database schema ( in my case, that's db2user1 ) resulting in a pair of SQL scripts that can then be executed against a new database schema ( in my case, that's db2user2 ) to migrate the roles AND data from one to t'other.

I was quite obviously doing something wrong, as I kept seeing exceptions such as: -

ilog.rules.teamserver.model.IlrConnectException
Caused by: java.lang.NullPointerException


in the Decision Center logs post migration.

It was, however, a PEBKAC - I was obviously doing it wrong.

Once I changed my approach: -

Restore backup of RTSDB - as db2inst1

db2 restore database RTSDB from /tmp

Connect as the instance owner - as db2inst1

db2 connect to RTSDB user db2inst1 using passw0rd

Grant the new schema - db2user2 - rights to the database - as db2inst1

db2 grant dbadm on database to user db2user2

Terminate the DB2 session - as db2inst1

db2 terminate

Connect as the new schema - as db2inst1

db2 connect to RTSDB user db2user2 using passw0rd

Run the provided script to create the necessary ODM tables, indices etc. - as db2inst1

db2 -tvf ~/rule-team-server.sql

- For the record, this SQL script is generated when one first logs into the Decision Center, so I just kept a copy :-)

Terminate the DB2 session - as db2inst1

db2 terminate

Start the Decision Centre cluster - as wasadmin

AdminControl.invoke('WebSphere:name=DecisionCenterCluster,process=dmgr,platform=common,node=Dmgr,version=8.5.5.11,type=Cluster,mbeanIdentifier=DecisionCenterCluster,cell=odmCell01,spec=1.0', 'start')

Log into the Decision Centre - as wasadmin


Follow the prompts to complete basic setup - extensions, locale ( I'm using en_GB as I'm a Brit ) etc. *BUT* avoid executing the generated SQL ( this has already been done ), as per this: -


At this stage, we've got a clean Decision Center database, ready to receive BUT with no roles or data migrated from the old environment: -


Set variables - as wasadmin

export ANT_HOME=/opt/ibm/ODM89/shared/tools/ant
export PATH=$PATH:/opt/ibm/ODM89/shared/tools/ant/bin
export JAVA_HOME=/opt/ibm/ODM89/jdk/jre


Run the Ant scripts to extract the roles and data for the old db2user1 schema - as wasadmin

cd /opt/ibm/ODM89/teamserver/bin

ant -DoldDatabaseSchemaName=db2user1 -Dserver.url=https://odm.uk.ibm.com:9443/teamserver -DrtsAdmin.login=wasadmin -DrtsAdmin.password=passw0rd -DoutputFile=/tmp/migrate_roles.sql gen-migration80-role

ant -DoldDatabaseSchemaName=db2user1 -Dserver.url=https://odm.uk.ibm.com:9443/teamserver -DdatasourceName=jdbc/ilogDataSource -DrtsAdmin.login=wasadmin -DrtsAdmin.password=passw0rd -DoutputFile=/tmp/migrate_data.sql gen-migration80-script

This results in a pair of SQL scripts: -

-rw-r--r-- 1 wasadmin wasadmins 79746 Jul 16 16:25 /tmp/migrate_data.sql
-rw-r--r-- 1 wasadmin wasadmins  9552 Jul 16 16:24 /tmp/migrate_roles.sql


Connect as the new schema - as db2inst1

db2 connect to RTSDB user db2user2 using passw0rd

Execute migration scripts - as db2inst1

db2 -tvf /tmp/migrate_roles.sql
db2 -tvf /tmp/migrate_data.sql


Terminate the DB2 session - as db2inst1

db2 terminate

Log into the Decision Centre - as wasadmin


C'est voila, we have a NEW ODM 8.9 Decision Centre with the project that was previously created on the OLD ODM 8.0 Decision Centre: -



WebSphere Application Server - Transaction Logs - getting it wrong

I do need to write a long-form article about this, but I've been on a voyage of discovery configuring AND testing WAS transaction recovery, by placing the transaction/compensation/partner logs in an Oracle 12c database.

This is in the context of an IBM Business Process Manager Advanced environment.

During the process, I saw this in the SupCluster logs ( specifically the second cluster member ) : -

SupClusterMember2/SystemOut.log:[16/07/17 11:53:47:332 BST] 00000001 WASSessionCor I SessionProperties shouldSetAndDoLogging SESN0169I: Session Manager found the custom property InvalidateOnUnauthorizedSessionRequestException with value true.
SupClusterMember2/SystemOut.log:[16/07/17 11:59:40:316 BST] 0000004d SQLMultiScope I   CWRLS0009E: Details of recovery log failure: Another server has locked the HA lock row, com.ibm.ws.recoverylog.spi.InternalLogException: Another server has locked the HA lock row
SupClusterMember2/SystemOut.log:[16/07/17 11:59:40:319 BST] 0000004d SQLMultiScope E   CWRLS0024E: Exception caught during recovery! Another server has locked the HA lock row, com.ibm.ws.recoverylog.spi.InternalLogException: Another server has locked the HA lock row
SupClusterMember2/SystemOut.log:[16/07/17 11:59:40:324 BST] 0000004d SQLMultiScope A   WTRN0107W: Caught non-SQLException Throwable when forcing SQL RecoveryLog tranlog for server PSCell1\Node2\SupClusterMember2 Throwable: Another server has locked the HA lock row, com.ibm.ws.recoverylog.spi.InternalLogException: Another server has locked the HA lock row
SupClusterMember2/SystemOut.log:[16/07/17 11:59:40:333 BST] 0000004d SQLMultiScope A   WTRN0100E: Cannot recover from SQLException when forcing SQL RecoveryLog tranlog for server PSCell1\Node2\SupClusterMember2 Exception: Another server has locked the HA lock row, com.ibm.ws.recoverylog.spi.InternalLogException: Another server has locked the HA lock row


The problem was a PEBCAK, in that I'd obviously misconfigured things.

I validated my WAS configuration: -

cat /opt/ibm/WebSphereProfiles/Dmgr01/config/cells/PSCell1/nodes/Node1/serverindex.xml | grep -i recoveryLog

    <recoveryLog xmi:id="RecoveryLog_1500062274540" transactionLogDirectory="custom://com.ibm.rls.jdbc.SQLRecoveryLog?datasource=jdbc/AppCluster_Tranlogs,tablesuffix=App1" compensationLogDirectory="custom://com.ibm.rls.jdbc.SQLRecoveryLog?datasource=jdbc/AppCluster_Tranlogs,tablesuffix=App1" compensationLogFileSize="5"/>
    <recoveryLog xmi:id="RecoveryLog_1500062279192" transactionLogDirectory="custom://com.ibm.rls.jdbc.SQLRecoveryLog?datasource=jdbc/SupCluster_Tranlogs,tablesuffix=Sup1"/>

cat /opt/ibm/WebSphereProfiles/Dmgr01/config/cells/PSCell1/nodes/Node2/serverindex.xml | grep -i recoveryLog

    <recoveryLog xmi:id="RecoveryLog_1500062274432" transactionLogDirectory="custom://com.ibm.rls.jdbc.SQLRecoveryLog?datasource=jdbc/AppCluster_Tranlogs,tablesuffix=App2" compensationLogDirectory="custom://com.ibm.rls.jdbc.SQLRecoveryLog?datasource=jdbc/AppCluster_Tranlogs,tablesuffix=App2" compensationLogFileSize="5"/>
    <recoveryLog xmi:id="RecoveryLog_1500062279152" transactionLogDirectory="custom://com.ibm.rls.jdbc.SQLRecoveryLog?datasource=jdbc/SupCluster_Tranlogs,tablesuffix=Sup2"/>


to ensure that I : -

(a) was using the right datasources for the right clusters ( AppCluster has both Transaction and Compensation logs, whereas SupCluster only has Transaction logs )
(b) had suitably incremented the suffix - App1 or Sup1 for member 1, App2 or Sup2 for member 2

Finally, as this was a TEST environment, I dropped the tables: -

DROP TABLE CMNUSER.WAS_TRAN_LOGAPP1;
DROP TABLE CMNUSER.WAS_PARTNER_LOGAPP1;
DROP TABLE CMNUSER.WAS_COMP_LOGAPP1;
DROP TABLE CMNUSER.WAS_TRAN_LOGAPP2;
DROP TABLE CMNUSER.WAS_PARTNER_LOGAPP2;
DROP TABLE CMNUSER.WAS_COMP_LOGAPP2;
DROP TABLE CMNUSER.WAS_TRAN_LOGSUP1;
DROP TABLE CMNUSER.WAS_PARTNER_LOGSUP1;
DROP TABLE CMNUSER.WAS_COMP_LOGSUP1;
DROP TABLE CMNUSER.WAS_TRAN_LOGSUP2;
DROP TABLE CMNUSER.WAS_PARTNER_LOGSUP2;
DROP TABLE CMNUSER.WAS_COMP_LOGSUP2;


and recreated them: -

CREATE TABLE CMNUSER.WAS_TRAN_LOGAPP1(
 SERVER_NAME VARCHAR(128),
 SERVICE_ID SMALLINT,
 RU_ID NUMBER(19),
 RUSECTION_ID NUMBER(19),
 RUSECTION_DATA_INDEX SMALLINT,
 DATA BLOB);

CREATE TABLE CMNUSER.WAS_PARTNER_LOGAPP1(
 SERVER_NAME VARCHAR(128),
 SERVICE_ID SMALLINT,
 RU_ID NUMBER(19),
 RUSECTION_ID NUMBER(19),
 RUSECTION_DATA_INDEX SMALLINT,
 DATA BLOB);

CREATE TABLE CMNUSER.WAS_COMP_LOGAPP1(
 SERVER_NAME VARCHAR(128),
 SERVICE_ID SMALLINT,
 RU_ID NUMBER(19),
 RUSECTION_ID NUMBER(19),
 RUSECTION_DATA_INDEX SMALLINT,
 DATA BLOB);

 CREATE TABLE CMNUSER.WAS_TRAN_LOGAPP2(
 SERVER_NAME VARCHAR(128),
 SERVICE_ID SMALLINT,
 RU_ID NUMBER(19),
 RUSECTION_ID NUMBER(19),
 RUSECTION_DATA_INDEX SMALLINT,
 DATA BLOB);

CREATE TABLE CMNUSER.WAS_PARTNER_LOGAPP2(
 SERVER_NAME VARCHAR(128),
 SERVICE_ID SMALLINT,
 RU_ID NUMBER(19),
 RUSECTION_ID NUMBER(19),
 RUSECTION_DATA_INDEX SMALLINT,
 DATA BLOB);

CREATE TABLE CMNUSER.WAS_COMP_LOGAPP2(
 SERVER_NAME VARCHAR(128),
 SERVICE_ID SMALLINT,
 RU_ID NUMBER(19),
 RUSECTION_ID NUMBER(19),
 RUSECTION_DATA_INDEX SMALLINT,
 DATA BLOB);

 CREATE TABLE CMNUSER.WAS_TRAN_LOGSUP1(
  SERVER_NAME VARCHAR(128),
  SERVICE_ID SMALLINT,
  RU_ID NUMBER(19),
  RUSECTION_ID NUMBER(19),
  RUSECTION_DATA_INDEX SMALLINT,
  DATA BLOB);

 CREATE TABLE CMNUSER.WAS_PARTNER_LOGSUP1(
  SERVER_NAME VARCHAR(128),
  SERVICE_ID SMALLINT,
  RU_ID NUMBER(19),
  RUSECTION_ID NUMBER(19),
  RUSECTION_DATA_INDEX SMALLINT,
  DATA BLOB);

 CREATE TABLE CMNUSER.WAS_COMP_LOGSUP1(
  SERVER_NAME VARCHAR(128),
  SERVICE_ID SMALLINT,
  RU_ID NUMBER(19),
  RUSECTION_ID NUMBER(19),
  RUSECTION_DATA_INDEX SMALLINT,
  DATA BLOB);

  CREATE TABLE CMNUSER.WAS_TRAN_LOGSUP2(
   SERVER_NAME VARCHAR(128),
   SERVICE_ID SMALLINT,
   RU_ID NUMBER(19),
   RUSECTION_ID NUMBER(19),
   RUSECTION_DATA_INDEX SMALLINT,
   DATA BLOB);

  CREATE TABLE CMNUSER.WAS_PARTNER_LOGSUP2(
   SERVER_NAME VARCHAR(128),
   SERVICE_ID SMALLINT,
   RU_ID NUMBER(19),
   RUSECTION_ID NUMBER(19),
   RUSECTION_DATA_INDEX SMALLINT,
   DATA BLOB);

  CREATE TABLE CMNUSER.WAS_COMP_LOGSUP2(
   SERVER_NAME VARCHAR(128),
   SERVICE_ID SMALLINT,
   RU_ID NUMBER(19),
   RUSECTION_ID NUMBER(19),
   RUSECTION_DATA_INDEX SMALLINT,
   DATA BLOB);

And all is well.

From a testing perspective, I've created a SCA module which uses a JDBC Resource Adapter to create/update/read data from an Oracle database table.

Again, that's for a future long-form article ….

XAException occurred. Error code is: XAER_RMERR (-3). Exception is:

I'm tinkering with WebSphere Application Server, underlying IBM BPM Advanced, in the context of transaction/compensation/partner log recovery.

In this scenario, I'm placing these logs into a database, Oracle 12c.

It's been a useful - and interesting - learning curve, and I've pretty much proved/tested the concept.

I did, however, see one issue today; specifically, these messages: -

[16/07/17 13:34:06:748 BST] 00000074 XARecoveryDat A   WTRN0151I: Preparing to call xa recover on XAResource: dave
[16/07/17 13:34:06:866 BST] 00000074 WSRdbXaResour E   DSRA0304E:  XAException occurred. XAException contents and details are: "".
[16/07/17 13:34:06:866 BST] 00000074 WSRdbXaResour E   DSRA0302E:  XAException occurred.  Error code is: XAER_RMERR (-3).  Exception is: <null>
[16/07/17 13:34:06:866 BST] 00000074 XARminst      E   WTRN0037W: The transaction service encountered an error on an xa_recover operation. The resource was com.ibm.ws.rsadapter.spi.WSRdbXaResourceImpl@35d8b386. The error code was XAER_RMERR. The exception stack trace follows: javax.transaction.xa.XAException
at oracle.jdbc.xa.OracleXAResource.recover(OracleXAResource.java:727)
at com.ibm.ws.rsadapter.spi.WSRdbXaResourceImpl.recover(WSRdbXaResourceImpl.java:1111)
at com.ibm.ws.Transaction.JTA.XARminst.recover(XARminst.java:141)
at com.ibm.ws.Transaction.JTA.XARecoveryData.recover(XARecoveryData.java:723)
at com.ibm.tx.jta.impl.PartnerLogTable.recover(PartnerLogTable.java:432)
at com.ibm.tx.jta.impl.RecoveryManager.resync(RecoveryManager.java:1544)
at com.ibm.tx.jta.impl.RecoveryManager.performResync(RecoveryManager.java:2286)
at com.ibm.ws.tx.jta.RecoveryManager.performResync(RecoveryManager.java:119)
at com.ibm.tx.jta.impl.RecoveryManager.run(RecoveryManager.java:2239)
at java.lang.Thread.run(Thread.java:809)


appearing in the SystemOut.log for one of my cluster members - AppCluster.

This IBM Knowledge Center article: -


<snip>
Problem

When WebSphere Application Server attempts to recover Oracle database transactions, the transaction service issues the following exception:

WTRN0037W: The transaction service encountered an error on an xa_recover operation. 
The resource was com.ibm.ws.rsadapter.spi.WSRdbXaResourceImpl@1114a62. 
The error code was XAER_RMERR. The exception stack trace follows: 
javax.transaction.xa.XAException
at oracle.jdbc.xa.OracleXAResource.recover(OracleXAResource.java:726)
at com.ibm.ws.rsadapter.spi.WSRdbXaResourceImpl.recover(WSRdbXaResourceImpl.java:954)
at com.ibm.ws.Transaction.JTA.XARminst.recover(XARminst.java:137)
at com.ibm.ws.Transaction.JTA.XARecoveryData.recover(XARecoveryData.java:609)
at com.ibm.ws.Transaction.JTA.PartnerLogTable.recover(PartnerLogTable.java:511)
at com.ibm.ws.Transaction.JTA.RecoveryManager.resync(RecoveryManager.java:1784)
at com.ibm.ws.Transaction.JTA.RecoveryManager.run(RecoveryManager.java:2241)

Cause

Oracle requires services such as the WebSphere Application Server transaction service to have special permissions for performing transaction recoveries.

Solution

As user SYS, run the following commands on your Oracle server:

grant select on pending_trans$ to public;
grant select on dba_2pc_pending to public;
grant select on dba_pending_transactions to public;
grant execute on dbms_system to <user>;

User is a user ID in the application server that is authorized to perform transaction recovery for the XA data source. If you have not authorized any user IDs to perform transaction recovery, the application server uses the login alias for the data source as the user ID.
This problem is mentioned under Oracle bug: 3979190. Running the preceding commands solves the problem.
</snip>

Following the KC article, I updated the Oracle DB permissions: -

sqlplus / as sysdba

SQL*Plus: Release 12.2.0.1.0 Production on Sun Jul 16 13:20:39 2017

Copyright (c) 1982, 2016, Oracle.  All rights reserved.

Connected to:
Oracle Database 12c Standard Edition Release 12.2.0.1.0 - 64bit Production


grant select on pending_trans$ to public;

Grant succeeded.

grant select on dba_2pc_pending to public;

Grant succeeded.

grant select on dba_pending_transactions to public;

Grant succeeded.

grant execute on dbms_system to cmnuser;

Grant succeeded.

and then restarted the AppCluster.

Let's see how it goes, but it's looking good so far.

Friday, 14 July 2017

TypeError: __add__ nor __radd__ defined for these operands

Whilst scripting the configuration of the WebSphere Application Server transaction/compensation/partner logs into an Oracle database: -

/opt/ibm/WebSphereProfiles/Dmgr01/bin/wsadmin.sh -lang jython -user wasadmin -password passw0rd -f tranlogs.jy 

I saw this: -

WASX7209I: Connected to process "dmgr" on node Dmgr using SOAP connector;  The type of process is: DeploymentManager
WASX7017E: Exception received while running file "tranlogs.jy"; exception information: com.ibm.bsf.BSFException: exception from Jython:
Traceback (innermost last):
  File "<string>", line 18, in ?
TypeError: __add__ nor __radd__ defined for these operands


Last time I saw this error ( 2012 ), it was user error.

I checked my script ( specifically line 18 as per the exception ) : -

datasource=AdminTask.createDatasource(provider, '[-name '+clusterName+'_Tranlogs -jndiName jdbc/'+clusterName+'_Tranlogs -dataStoreHelperClassName com.ibm.websphere.rsadapter.Oracle11gDataStoreHelper -containerManagedPersistence true -componentManagedAuthenticationAlias '+j2cAlias+' -configureResourceProperties [[URL java.lang.String jdbc:oracle:thin:@//'+dbHost+':'+dbPort+'/'+dbName+']]]')

Looking at the concatenation of three strings ( dbHost, dbPort and dbName ) gave me a clue, and I checked the variables, and found: -

dbPort=1521

In other words, I'd defined it as an integer whereas Jython was expected a string.

Once I changed it: -

dbPort='1521'

the script ran without errors.

CWMCB0012W: An inconsistent configuration has been detected for data source jdbc/TeamWorksDB

This follows on from an earlier post: -


Having updated my JDBC data sources to reflect the correct Oracle service name: -


I'm now seeing the following warning: -

[14/07/17 20:07:17:855 BST] 000000f5 ConfigReader  W com.ibm.bpm.config.util.ws.ConfigReader getDatabaseInfo CWMCB0012W: An inconsistent configuration has been detected for data source jdbc/TeamWorksDB. It is assumed that the correct Oracle service name or SID is orcl.uk.ibm.com rather than orcl.

in the SystemOut.log for the AppCluster members.

The IBM BPM Knowledge Center has a solution: -

CWMCB0012W   An inconsistent configuration has been detected for data source 0. It is assumed that the correct Oracle service name or SID is 1 rather than 2.

Explanation

An IBM BPM data source has been manually modified rather than updated using the "BPMConfig -update -dataSource" command. The resulting configuration is inconsistent.
User response

Export the IBM BPM deployment environment configuration and fix the data source configuration by running the "BPMConfig -update -dataSource" command.

Following the advice therein, I exported the BPM configuration: -

/opt/ibm/WebSphere/AppServer/bin/BPMConfig.sh -export -profile Dmgr01 -de De1 -outputDir /tmp/De1

Logging to file /opt/ibm/WebSphere/AppServer/logs/config/BPMConfig_20170714-203920.log.
CWMCB0012W: An inconsistent configuration has been detected for data source jdbc/WPSDB. It is assumed that the correct Oracle service name or SID is orcl.uk.ibm.com rather than orcl.
CWMCB0012W: An inconsistent configuration has been detected for data source jdbc/ECMDBXA. It is assumed that the correct Oracle service name or SID is orcl.uk.ibm.com rather than orcl.
CWMCB0012W: An inconsistent configuration has been detected for data source jdbc/CommonDB. It is assumed that the correct Oracle service name or SID is orcl.uk.ibm.com rather than orcl.
CWMCB0012W: An inconsistent configuration has been detected for data source jdbc/BPEDB. It is assumed that the correct Oracle service name or SID is orcl.uk.ibm.com rather than orcl.
CWMCB0012W: An inconsistent configuration has been detected for data source jdbc/ECMDB. It is assumed that the correct Oracle service name or SID is orcl.uk.ibm.com rather than orcl.
CWMCB0012W: An inconsistent configuration has been detected for data source jdbc/SharedDb. It is assumed that the correct Oracle service name or SID is orcl.uk.ibm.com rather than orcl.
The 'BPMConfig.sh -export -profile Dmgr01 -de De1 -outputDir /tmp/De1' command completed successfully.

and then updated the Data Source configuration: -

/opt/ibm/WebSphere/AppServer/bin/BPMConfig.sh -update -dataSource /tmp/De1/De1.properties 

Logging to file /opt/ibm/WebSphere/AppServer/logs/config/BPMConfig_20170714-204136.log.
Saving configuration changes...
The 'BPMConfig.sh -update -dataSource /tmp/De1/De1.properties' command completed successfully.


and then re-exported the configuration to check that all was good: -

/opt/ibm/WebSphere/AppServer/bin/BPMConfig.sh -export -profile Dmgr01 -de De1 -outputDir /tmp/Foobar

Logging to file /opt/ibm/WebSphere/AppServer/logs/config/BPMConfig_20170714-204220.log.
The 'BPMConfig.sh -export -profile Dmgr01 -de De1 -outputDir /tmp/Foobar' command completed successfully.


So that's all good then :-)

IBM BPM - "CWSCA8095W: Unable to find class..." seen during serviceDeploy process

I'm tinkering with IBM BPM Advanced again, specifically taking an export of an SCA module, built using IBM Integration Designer and exported as a Project Interchange (PI) file, and attempting to "compile" it ready for deploying to an external IBM BPM 8.57 environment.

Having exported the PI file, and made it available to the BPM box, I then ran serviceDeploy.sh against it: -

/opt/ibm/WebSphere/AppServer/bin/serviceDeploy.sh OracleTest.zip 

which returned: -

The workbench is starting in /home/wasadmin/15d428ea0a1.
The workspace is initializing.
OracleTest.zip was successfully imported into the workspace.
The OracleTestApp project is being created.
The OracleTestWeb project is being created.
The OracleTest project is building.
OracleTest is being validated.

The following 36 warning messages were reported during deployment:

Severity:  [warning]
Description:  CWSCA8095W: Unable to find class com.ibm.j2ca.jdbc.JDBCInteractionSpec, please make sure the class path is properly set up.
Resource:  toAndFromOracle.import
Location:        

Severity:  [warning]
Description:  CWSCA8095W: Unable to find class com.ibm.j2ca.jdbc.JDBCInteractionSpec, please make sure the class path is properly set up.
Resource:  toAndFromOracle.import
Location:        

Severity:  [warning]
Description:  CWSCA8095W: Unable to find class com.ibm.j2ca.jdbc.JDBCInteractionSpec, please make sure the class path is properly set up.
Resource:  toAndFromOracle.import
Location:        

Severity:  [warning]
Description:  CWSCA8095W: Unable to find class com.ibm.j2ca.extension.emd.runtime.WBIFaultSelectorImpl, please make sure the class path is properly set up.
Resource:  toAndFromOracle.import
Location:        

Severity:  [warning]
Description:  CWSCA8085W: The fault selector of com.ibm.j2ca.extension.emd.runtime.WBIFaultSelectorImpl type is missing in the class path or in the dependencies.
Resource:  toAndFromOracle.import
Location:        

Severity:  [warning]
Description:  CWSCA8095W: Unable to find class com.ibm.j2ca.extension.emd.runtime.WBIFaultDataBindingImpl, please make sure the class path is properly set up.
Resource:  toAndFromOracle.import
Location:        

Severity:  [warning]
Description:  CWSCA9537W: The fault binding of com.ibm.j2ca.extension.emd.runtime.WBIFaultDataBindingImpl type is missing in the class path or in the dependencies.
Resource:  toAndFromOracle.import
Location:        

Severity:  [warning]
Description:  CWSCA8095W: Unable to find class com.ibm.j2ca.extension.emd.runtime.WBIFaultDataBindingImpl, please make sure the class path is properly set up.
Resource:  toAndFromOracle.import
Location:        

Severity:  [warning]
Description:  CWSCA9537W: The fault binding of com.ibm.j2ca.extension.emd.runtime.WBIFaultDataBindingImpl type is missing in the class path or in the dependencies.
Resource:  toAndFromOracle.import
Location:        

Severity:  [warning]
Description:  CWSCA8095W: Unable to find class com.ibm.j2ca.extension.emd.runtime.WBIFaultDataBindingImpl, please make sure the class path is properly set up.
Resource:  toAndFromOracle.import
Location:        

Severity:  [warning]
Description:  CWSCA9537W: The fault binding of com.ibm.j2ca.extension.emd.runtime.WBIFaultDataBindingImpl type is missing in the class path or in the dependencies.
Resource:  toAndFromOracle.import
Location:        

Severity:  [warning]
Description:  CWSCA8095W: Unable to find class com.ibm.j2ca.extension.emd.runtime.WBIFaultDataBindingImpl, please make sure the class path is properly set up.
Resource:  toAndFromOracle.import
Location:        

Severity:  [warning]
Description:  CWSCA9537W: The fault binding of com.ibm.j2ca.extension.emd.runtime.WBIFaultDataBindingImpl type is missing in the class path or in the dependencies.
Resource:  toAndFromOracle.import
Location:        

Severity:  [warning]
Description:  CWSCA8095W: Unable to find class com.ibm.j2ca.extension.emd.runtime.WBIFaultDataBindingImpl, please make sure the class path is properly set up.
Resource:  toAndFromOracle.import
Location:        

Severity:  [warning]
Description:  CWSCA9537W: The fault binding of com.ibm.j2ca.extension.emd.runtime.WBIFaultDataBindingImpl type is missing in the class path or in the dependencies.
Resource:  toAndFromOracle.import
Location:        

Severity:  [warning]
Description:  CWSCA8095W: Unable to find class com.ibm.j2ca.extension.emd.runtime.WBIFaultDataBindingImpl, please make sure the class path is properly set up.
Resource:  toAndFromOracle.import
Location:        

Severity:  [warning]
Description:  CWSCA9537W: The fault binding of com.ibm.j2ca.extension.emd.runtime.WBIFaultDataBindingImpl type is missing in the class path or in the dependencies.
Resource:  toAndFromOracle.import
Location:        

Severity:  [warning]
Description:  CWSCA8095W: Unable to find class com.ibm.j2ca.extension.emd.runtime.WBIFaultDataBindingImpl, please make sure the class path is properly set up.
Resource:  toAndFromOracle.import
Location:        

Severity:  [warning]
Description:  CWSCA9537W: The fault binding of com.ibm.j2ca.extension.emd.runtime.WBIFaultDataBindingImpl type is missing in the class path or in the dependencies.
Resource:  toAndFromOracle.import
Location:        

Severity:  [warning]
Description:  CWSCA8095W: Unable to find class com.ibm.j2ca.extension.emd.runtime.MatchingFaultDataBinding, please make sure the class path is properly set up.
Resource:  toAndFromOracle.import
Location:        

Severity:  [warning]
Description:  CWSCA9537W: The fault binding of com.ibm.j2ca.extension.emd.runtime.MatchingFaultDataBinding type is missing in the class path or in the dependencies.
Resource:  toAndFromOracle.import
Location:        

Severity:  [warning]
Description:  CWSCA8095W: Unable to find class com.ibm.j2ca.extension.emd.runtime.WBIFaultDataBindingImpl, please make sure the class path is properly set up.
Resource:  toAndFromOracle.import
Location:        

Severity:  [warning]
Description:  CWSCA9537W: The fault binding of com.ibm.j2ca.extension.emd.runtime.WBIFaultDataBindingImpl type is missing in the class path or in the dependencies.
Resource:  toAndFromOracle.import
Location:        

Severity:  [warning]
Description:  CWSCA8095W: Unable to find class com.ibm.j2ca.extension.emd.runtime.WBIFaultDataBindingImpl, please make sure the class path is properly set up.
Resource:  toAndFromOracle.import
Location:        

Severity:  [warning]
Description:  CWSCA9537W: The fault binding of com.ibm.j2ca.extension.emd.runtime.WBIFaultDataBindingImpl type is missing in the class path or in the dependencies.
Resource:  toAndFromOracle.import
Location:        

Severity:  [warning]
Description:  CWSCA8095W: Unable to find class com.ibm.j2ca.extension.emd.runtime.WBIFaultDataBindingImpl, please make sure the class path is properly set up.
Resource:  toAndFromOracle.import
Location:        

Severity:  [warning]
Description:  CWSCA9537W: The fault binding of com.ibm.j2ca.extension.emd.runtime.WBIFaultDataBindingImpl type is missing in the class path or in the dependencies.
Resource:  toAndFromOracle.import
Location:        

Severity:  [warning]
Description:  CWSCA8095W: Unable to find class com.ibm.j2ca.extension.emd.runtime.MatchingFaultDataBinding, please make sure the class path is properly set up.
Resource:  toAndFromOracle.import
Location:        

Severity:  [warning]
Description:  CWSCA9537W: The fault binding of com.ibm.j2ca.extension.emd.runtime.MatchingFaultDataBinding type is missing in the class path or in the dependencies.
Resource:  toAndFromOracle.import
Location:        

Severity:  [warning]
Description:  CWSCA8095W: Unable to find class com.ibm.j2ca.extension.emd.runtime.WBIFaultDataBindingImpl, please make sure the class path is properly set up.
Resource:  toAndFromOracle.import
Location:        

Severity:  [warning]
Description:  CWSCA9537W: The fault binding of com.ibm.j2ca.extension.emd.runtime.WBIFaultDataBindingImpl type is missing in the class path or in the dependencies.
Resource:  toAndFromOracle.import
Location:        

Severity:  [warning]
Description:  CWSCA8095W: Unable to find class com.ibm.j2ca.jdbc.JDBCManagedConnectionFactory, please make sure the class path is properly set up.
Resource:  toAndFromOracle.import
Location:        

Severity:  [warning]
Description:  CWSCA8043W: com.ibm.j2ca.jdbc.JDBCManagedConnectionFactory cannot be validated.
Resource:  toAndFromOracle.import
Location:        

Severity:  [warning]
Description:  CWSCA8095W: Unable to find class com.ibm.j2ca.jdbc.JDBCResourceAdapter, please make sure the class path is properly set up.
Resource:  toAndFromOracle.import
Location:        

Severity:  [warning]
Description:  CWSCA8082E: The resource adapter of the com.ibm.j2ca.jdbc.JDBCResourceAdapter type is missing in the class path or in the dependencies.
Resource:  toAndFromOracle.import
Location:        

Severity:  [warning]
Description:  XSD: The location 'JDBCASI.xsd' has not been resolved
Resource:  DaveConsultant.xsd
Location:       1


The OracleTestApp.ear application is being exported.
Deployment has completed.
The workspace is being deleted.


Now I've definitely done this before ….

… but, this time around, I'm also using a JDBC Resource Adapter (RA) as evidenced by the messages e.g.

CWSCA8082E: The resource adapter of the com.ibm.j2ca.jdbc.JDBCResourceAdapter type is missing in the class path or in the dependencies.

This provided a clue.

Next time around, I exported BOTH the SCA module AND the dependent project that contains the JDBC Resource Adapter code ( this was created for me when I dragged a JDBC Adapter component into my Assembly Diagram : -


For the record, this was the PI file first time around: -

-rwxr-xr-x   1 wasadmin wasadmins   31538 Jul 14 20:23 OracleTest.zip

and this was the version the second time around: -

-rwxr-xr-x   1 wasadmin wasadmins 5747285 Jul 14 20:30 OracleTest.zip

It looks slightly more podgy because it contains the RA code.

More importantly, the "compilation" process worked a treat: -

The workbench is starting in /home/wasadmin/15d42955f45.
The workspace is initializing.
OracleTest.zip was successfully imported into the workspace.
The OracleTestApp project is being created.
The OracleTestWeb project is being created.
The OracleTest project is building.
OracleTest is being validated.
The OracleTestApp.ear application is being exported.
Deployment has completed.
The workspace is being deleted.