Friday, 19 September 2014

WebSphere Application Server - Recovering from failed transaction recovery

Thanks to Twitter for sharing: -

Great blog on Recovering from failed transaction recovery. Very helpful! Take a look! #wasserv
19/09/2014 17:10

When WebSphere Application Server is running a transaction, the transaction information is written to the tranlog directory to log1 & log2. The resources required for that transaction (database name, user, password, etc) are recorded in the partnerlog directory to it's log1 & log2. When a transaction completes, the transaction information is garbage collected from the logs. If the application server should abend or be forced off mid-transaction, such that a transaction does not complete, then on subsequent server restarts, the transaction service detects the unfinished transaction and attempts to re-establish the resource(s) stored in the partnerlogs then complete the transaction(s) stored in the travelogs.

IBM Integration Bus - The Fun Continues .... MQ reason code 2035 while trying to connect

So I now have IBM Integration Bus (IIB) and the Toolkit running on my Red Hat Enterprise Linux VM, as per these most recent posts: -

I have the toolkit installed as user wasadmin because that's the user with which I installed a whole slew of other WebSphere products, using IBM Installation Manager.

Therefore, in order to use the Toolkit to create/administer an Integration Node ( fka Broker ), I need to ensure that the wasadmin user is setup to access IIB and WebSphere MQ (WMQ).

I did this by: -

(a) Adding the user into the appropriate Linux groups - this is what I now have: -

id wasadmin

uid=500(wasadmin) gid=505(mqm) groups=505(mqm),506(mqbrkrs)

id mqm

uid=505(mqm) gid=505(mqm) groups=505(mqm)

id wmbadmin

uid=506(wmbadmin) gid=506(mqbrkrs) groups=506(mqbrkrs),505(mqm)

In essence, I added the wasadmin user to the mqbrkrs group as follows: -

usermod -G mqm,mqbrkrs wasadmin

( as root )

(b) configuring the Bash environment for user wasadmin to use BOTH WMQ and IIB: -

cat /home/wasadmin/.bashrc

# .bashrc

# Source global definitions
if [ -f /etc/bashrc ]; then
. /etc/bashrc

# User specific aliases and functions
umask 022
set -o vi
alias hist='history | cut -c 8-'

. /opt/ibm/mqm/bin/setmqenv -s -k

source /opt/ibm/mqsi/

I have highlighted in bold the two lines added to achieve this.

Now I can log in as wasadmin and create an Integration Node ( aka Broker ), as follows: -

mqsicreatebroker DAVEHAY -q DAVEHAY

WebSphere MQ queue manager created.
Directory '/var/mqm/qmgrs/DAVEHAY' created.
The queue manager is associated with installation 'Installation2'.
Creating or replacing default objects for queue manager 'DAVEHAY'.
Default objects statistics : 79 created. 0 replaced. 0 failed.
Completing setup.
Setup completed.
WebSphere MQ queue manager 'DAVEHAY' starting.
The queue manager is associated with installation 'Installation2'.
5 log records accessed on queue manager 'DAVEHAY' during the log replay phase.
Log replay for queue manager 'DAVEHAY' complete.
Transaction manager state recovered for queue manager 'DAVEHAY'.
WebSphere MQ queue manager 'DAVEHAY' started using V8.0.0.0.
BIP8071I: Successful command completion. 

mqsistart DAVEHAY

BIP8096I: Successful command initiation, check the system log to ensure that the component started without problem and that it continues to run without problem. 


BIP1285I: Broker 'DAVEHAY' on queue manager 'DAVEHAY' is stopped. 
BIP8071I: Successful command completion.


QMNAME(QM_MDB)                                            STATUS(Running)
QMNAME(DAVEHAY)                                           STATUS(Running)

I can then start the Toolkit: -


One small glitch - I see this within the Toolkit when I attempt to connect to the Integration Node: -

The user 'wasadmin' is not authorized to connect to queue manager 'DAVEHAY' (MQ reason code 2035 while trying to connect) 

Happily the solution is relatively simple :-)

I looked at the logs: -

cd /var/mqm/qmgrs/DAVEHAY/errors

wherein I could see: -

AMQ5534: User ID 'wasadmin' authentication failed


The user ID and password supplied by 'javaw' could not be authenticated.


Ensure that the correct user ID and password are provided by the application. Ensure that the authentication repository is correctly configured. Look at previous error messages for any additional information.

AMQ5542: The failed authentication check was caused by the queue manager CONNAUTH CHCKLOCL(OPTIONAL) configuration.


The user ID 'wasadmin' and its password were checked because the queue manager connection authority (CONNAUTH) configuration refers to an authentication information (AUTHINFO) object named 'SYSTEM.DEFAULT.AUTHINFO.IDPWOS' with CHCKLOCL(OPTIONAL).

This message accompanies a previous error to clarify the reason for the user ID and password check.

Refer to the previous error for more information.

Ensure that a password is specified by the client application and that the password is correct for the user ID. The authentication configuration of the queue manager connection determines the user ID repository. For example, the local operating system user database or an LDAP server.

If the CHCKCLNT setting is OPTIONAL, the authentication check can be avoided by not passing a user ID across the channel. For example, by omitting the MQCSP structure from the client MQCONNX API call.

To avoid the authentication check, you can amend the authentication configuration of the queue manager connection, but you should generally not allow unauthenticated remote access.

After some digging, I think that the problem is that the Toolkit is attempting to connect to the DAVEHAY Queue Manager, and is not sending the correct authentication string e.g. user ID *AND* password.

Looking at the Queue Manager in more detail: -

runmqsc DAVEHAY

5724-H72 (C) Copyright IBM Corp. 1994, 2014.
Starting MQSC for queue manager DAVEHAY.

display qmgr connauth

AMQ8408: Display Queue Manager details.


AMQ8566: Display authentication information details.
   AUTHTYPE(IDPWOS)                        ADOPTCTX(NO)
   DESCR( )                                CHCKCLNT(REQDADM)
   CHCKLOCL(OPTIONAL)                      FAILDLAY(1)
   ALTDATE(2014-09-19)                     ALTTIME(11.21.38)

My reading of this is: -

(a) the Toolkit is sending a user ID ( wasadmin ) but NOT sending a password
(b) the Queue Manager expects BOTH a user ID and a password

Interestingly, the Toolkit, leastways on Linux, doesn't give one the opportunity to specific connection credentials :-)

Therefore, the Toolkit, whilst running as wasadmin, is trying to connect to the Queue Manager as that user wit NO password :-)

Ironically, I'm making a LOCAL connection as both the Toolkit AND the Queue Manager are running on the same OS.

After much faffing about, I found this excellent deck from my IBM Hursley colleague, Morag Hughson: -

which led me to this solution: -


which, as far as I can establish, means that MQ will require a valid set of credentials for client connections, via CHCKCLNT(REQDADM), but not for local connections, via CHCKLOCL(NONE).

To verify this hypothesis, I switched back to the old state: -


and my old friend: -

The user 'wasadmin' is not authorized to connect to queue manager 'DAVEHAY' (MQ reason code 2035 while trying to connect) 

popped up.

I reverted back to CHCKLOCL(NONE) and I'm now in like Flynn: -

On a related note, I had a similar, but different problem with WMQ Explorer ( running as another user - wmbadmin ): -


With this, I was attempting to make a remote connection to the same Queue Manager: -

Host name or IP address
Port number 1414
Server-connection channel SYSTEM.DEF.SVRCONN

Unlike the IIB Toolkit, this DOES allow me to enter credentials.

However, Explorer failed to connect with: -

Could not establish a connection to the queue manager - reason 2538. (AMQ4059)
  Could not establish a connection to the queue manager - reason 2538. (AMQ4059)
  Severity: 10 (Warning)
  Explanation: The attempt to connect to the queue manager failed. This could be because the queue manager is incorrectly configured to allow a connection from this system, or the connection has been broken.
  Response: Try the operation again. If the error persists, examine the problem determination information to see if any information has been recorded.

I took the default port of 1414 so then checked to see whether it was running: -

netstat -aon | grep LISTENING | grep 1414

unix  2      [ ACC ]     STREAM     LISTENING     11460  /var/run/mcelog-client

In other words, nothing was listening on port 1414.

I did have a Channel defined: -


AMQ8414: Display Channel details.
   ALTDATE(2014-09-19)                     ALTTIME(11.21.38)
   CERTLABL( )                             COMPHDR(NONE)
   COMPMSG(NONE)                           DESCR( )
   DISCINT(0)                              HBINT(300)
   KAINT(AUTO)                             MAXINST(999999999)
   MAXINSTC(999999999)                     MAXMSGL(4194304)
   MCAUSER( )                              MONCHL(QMGR)
   RCVDATA( )                              RCVEXIT( )
   SCYDATA( )                              SCYEXIT( )
   SENDDATA( )                             SENDEXIT( )
   SHARECNV(10)                            SSLCAUTH(REQUIRED)
   SSLCIPH( )                              SSLPEER( )

which was running: -

display chstatus(*)

AMQ8417: Display Channel Status details.
   CONNAME(                  CURRENT
   STATUS(RUNNING)                         SUBSTATE(RECEIVE)

and a default Listener: -


AMQ8630: Display listener information details.
   TRPTYPE(TCP)                            PORT(0)
   IPADDR( )                               BACKLOG(0)
   DESCR( )                                ALTDATE(2014-09-19)

*BUT* the Listener was listening on .... port 0, which ain't ever gonna work :-)

Therefore, I needed to create a new Listener: -


AMQ8626: WebSphere MQ listener created.

and then start it: -


AMQ8021: Request to start WebSphere MQ listener accepted.

Again, I'm in like Flynn: -

which is nice.

Ain't it sweet ?

Security Bulletin: A security vulnerability has been identified in Cognos BI Server shipped with IBM Business Monitor (CVE-2014-0107)

Security Bulletin: A security vulnerability has been identified in Cognos BI Server shipped with IBM Business Monitor (CVE-2014-0107)

IBM HTTP Server and SSL Signature Algorithms

So, whilst listening to this week's SecurityNow podcast, Episode 473 Google vs. SHA-1, I learned that Google plans to force the web to deprecate the SHA1 ( Secure Hash Algorithm ) from November 2014 even though Microsoft has a more moderate plan to move away from it by late 2017.

Google wants us to move to SHA2, aka SHA224 / SHA256 / SHA512, even though their own websites are still using SHA1 at the moment: -

Apparently, Google Chrome will start to provide visual feedback to end-users when they visit a site using SHA1, which has led to a bit of debate :-)

This led me to think about IBM HTTP Server, which I use and love on an almost daily basis.

I've created a lot of certificates in IHS using the IBM Global Security Toolkit ( GSK ), using syntax such as: -

/opt/IBM/HTTPServer/bin/gskcapicmd -cert -create -db /opt/IBM/HTTPServer/ssl/keystore.kdb -pw passw0rd -size 2048 -dn "\\,o=ibm\\,c=us\\" -label "" -default_cert yes

I was interested to see what hashing algorithm IHS uses by default.

Here's the answer: -

/opt/IBM/HTTPServer/bin/gskcapicmd -cert -list -db /opt/IBM/HTTPServer/ssl/keystore.kdb -pw passw0rd

Certificates found
* default, - personal, ! trusted

/opt/IBM/HTTPServer/bin/gskcapicmd -cert -details -db /opt/IBM/HTTPServer/ssl/keystore.kdb -pw passw0rd -label

Label :
Key Size : 2048
Version : X509 V3
Serial : 1f75f58469abb054
Issuer :\,o\=ibm\,c\=us
Subject :\,o\=ibm\,c\=us
Not Before : 29 August 2014 18:03:07 GMT+01:00
Not After : 30 August 2015 18:03:07 GMT+01:00
Public Key
    30 82 01 22 30 0D 06 09 2A 86 48 86 F7 0D 01 01
    01 05 00 03 82 01 0F 00 30 82 01 0A 02 82 01 01
    00 CB CB F2 27 8F 1B 50 E3 A4 9C D9 D4 4E BE 2E
    87 95 FC FF D3 23 01 39 7F 9B 11 1F 9F 91 4F 19
    61 3F 1E 2B B5 79 01 2B 04 A0 91 1F DF 68 22 85
    88 B4 76 B9 B9 FD 68 A0 D7 90 06 50 8E FA 0B 52
    96 14 A6 F3 A0 94 4C 63 41 04 89 F0 F5 0F 6E E0
    7E A4 A2 C9 AB 59 D1 0A 92 31 9D 20 A0 F5 A3 C6
    22 04 1E 30 71 7C D8 5D 86 82 D0 B9 91 8F 9F A3
    E2 FA 41 7D 57 06 FE 2C 5E 1F 9B 6F 77 18 25 22
    60 DE E8 84 59 CF E5 0E E4 90 5E 5F F8 A7 45 B9
    77 67 1C 3E CC 21 45 76 79 04 F5 2B D7 CA 86 1F
    95 3F D7 14 2A 90 21 25 AC 23 34 A2 05 99 DE 46
    C2 6A 19 BF 79 E3 EC 7C F8 BE C7 A1 DE EA 38 6B
    80 7C 92 21 38 5B 11 9B 7B E6 23 05 57 AD F8 68
    DA 21 3B 6D 2A FA 80 47 4D F4 1F 8F A0 FB 38 99
    0D D9 C9 B6 32 67 A5 E4 3F B4 11 E6 4C 98 4C 76
    FB BB 37 ED ED FC 9D 6F 23 D1 0D 7C 95 D3 B1 E9
    EF 02 03 01 00 01
Public Key Type : RSA (1.2.840.113549.1.1.1)
Fingerprint : SHA1 : 
    EA F8 A8 80 AC F6 86 29 66 48 A8 9F C2 73 23 99
    68 E8 3C 7D
Fingerprint : MD5 : 
    5A 7F 67 55 D7 0F D1 08 37 FE 6F 31 47 F9 DE F5
Fingerprint : SHA256 : 
    2B 05 82 F8 41 7E 9C 10 4B AE A8 99 18 DD 1D 7B
    50 56 F7 C6 16 5F A9 3F CE 07 19 A3 06 6F 13 24
Signature Algorithm : SHA1WithRSASignature (1.2.840.113549.1.1.5)
    88 3C 06 C5 DB FC A2 6C D4 C0 42 F4 1F A7 5D DF
    B7 FF B8 70 81 61 90 F5 D9 91 C4 9B E0 16 6A 61
    6C E5 55 C7 63 7E 9C 6A 05 6E C9 42 5C FA 26 4A
    6F 76 6F C5 6F 6E E5 E2 A4 65 8C BB 02 B1 A6 C4
    28 83 37 F1 39 BC 24 D1 9D 7F F2 66 95 5F 90 8E
    45 8E 97 95 61 89 C3 70 69 35 DC 2A CD FE E8 0A
    1A 8B 19 15 A3 DF BB 17 A5 A2 84 09 4F 32 12 47
    46 9D A4 16 F9 B4 E0 73 10 35 0B 0B AB EC 55 59
    00 DD F7 DA B7 44 DE 52 AB BE B3 B3 F5 40 5F 75
    FB 43 8E 4A FA 65 81 99 BB 97 7F DE 9B 88 8B ED
    11 14 FB 34 0B 15 6C EC 33 88 6F FB 41 AF 16 B0
    45 7A 41 2D 3C E4 B3 0B C8 56 81 B2 06 C9 C1 71
    D6 26 71 5C 13 61 39 B6 DD 97 8D 0D F4 84 34 9F
    3D F1 8B C2 E0 F8 11 2F 88 82 60 1B 50 A5 C0 97
    B0 5A 92 8B 1B DE D1 6A D3 BD 7B DC 7E AD 7A 2A
    EA DB 32 E3 14 95 69 94 9A D5 1B C2 A5 14 8F 7E
Trust Status : Enabled

So there's the answer, IHS generates a SHA1 certificate by default. I checked this for IHS 8.0 and 8.5, and the same is true in both cases.

Now that makes sense, given most of the world is still supporting SHA1 and, apparently, MS Windows XP SP2 doesn't support SHA2 - although I'd hope that any remaining Windows XP users would've moved to SP3 a looooong time ago; perhaps embedded Windows users don't have that luxury ?

Anyway, I then wondered whether I could've chosen to generate a SHA2 certificate using gskcapicmd. Guess what, I can :-)

Here's the multitude of choices: -

-Command usage-
-db | -crypto         Required
-tokenlabel           Required if -crypto present
-label                Required
-pw                   Optional
-dn                   Required
-type                 Optional if -db present <cms | kdb>
-expire               Optional
-size                 Optional
-x509version          Optional <1 | 2 | 3>
-default_cert         Optional <yes | no>
-ca                   Optional <true | false>
-sig_alg | -sigalg    Optional < | md5 | MD5_WITH_RSA | MD5WithRSA | sha1 | SHA_WITH_RSA | SHAWithRSA | SHA1WithRSA | sha224 | SHA224_WITH_RSA | SHA224WithRSA | sha256 | SHA256_WITH_RSA | SHA256WithRSA | SHA2WithRSA | sha384 | SHA384_WITH_RSA | SHA384WithRSA | SHA3WithRSA | sha512 | SHA512_WITH_RSA | SHA512WithRSA | SHA5WithRSA | SHA1WithECDSA | EC_ecdsa_with_SHA1 | SHA224WithECDSA | EC_ecdsa_with_SHA224 | SHA256WithECDSA | EC_ecdsa_with_SHA256 | SHA384WithECDSA | EC_ecdsa_with_SHA384 | SHA512WithECDSA | EC_ecdsa_with_SHA512>
-ca_label             Optional
-san_dnsname          Optional
-san_emailaddr        Optional
-san_ipaddr           Optional
-certpolicy           Optional
-eku                  Optional <ocspSigning, timeStamping, emailProtection, codeSigning, clientAuth, serverAuth, SSLStepUpApproval, any>
-ku                   Optional <digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment, keyAgreement, keyCertSign, cRLSign, encipherOnly, decipherOnly>
-template             Optional
-secondarydb          Optional if -crypto present
-secondarydbpw        Optional if -secondarydb present
-secondarydbtype      Optional if -secondarydb present

That's a LOT of choice :-)

Of course, one would need to think about the consuming devices BEFORE moving to SHA2, even though Google appear to be telling the rest of the internet what to do .....

Bottom line, think about the Signature Algorithm when creating certificates, along with everything else :-)

Thursday, 18 September 2014

IBM BPM Advanced - PFS-0065 seen in context of Performance Data Warehouse

So, hot on the heels of my previous post: -

I was looking to replicate the same for Process Center.

Strangely, when I looked at my Performance Data Warehouse (PDW) database for the Process Center environment, I did NOT have the TASKS table, merely LSW_TASK, and couldn't work out what SQL was actually used to create the missing table.

Interestingly, whilst there are SQL scripts to create all of the BPM/PDW/CMN database tables, there's nothing for TASKS. It appears that this is created dynamically.

When I dug further, via the Performance Admin console ( ), I could see errors in the View Errors tab.

I dug further into the underlying DB2 database tables, specifically LSW_DATA_TRANSFER_ERRORS : -

db2 "select error from db2user1.lsw_data_transfer_errors"

(PFS-0065) Snapshot 2064.8961f2a9-d0f9-419f-bfee-8e6437c4c0ba, undefined tracking group with external ID guid:06fb68640f875312:27f9b3f:148179f00e7:-7ffc. Send definitions to define the tracking group.

com.lombardisoftware.client.delegate.BusinessDelegateException: (PFS-0065) Snapshot 2064.8961f2a9-d0f9-419f-bfee-8e6437c4c0ba, undefined tracking group with external ID guid:06fb68640f875312:27f9b3f:148179f00e7:-7ffc. Send definitions to define the tracking group.

In other words, I was executing a BPD on the in-built Process Server run-time for which there was no corresponding Tracking Group. The clue is in the error above, helpfully highlighted in red.

This was easily resolved - I logged into the Process Admin console ( ), clicked on the Installed Apps tab and, for the app in question, clicked on the Update Tracking Definitions button: -

Once I did this, the TASKS table magically created itself, LSW_DATA_TRANSFER_ERRORS cleared itself down, and I was able to see statistics from my BPD within the TASKS table.

Now to go and turn off PDW so I never see statistics in the TASKS table again :-)

IBM BPM Advanced - Disabling Process Server to Performance Data Warehouse communication

So, in order to disable the automatic publishing of events from Process Server to the Performance Data Warehouse database, I followed this IBM Technote: -

My requirement is to disable the use of PDW and instead use IBM Business Monitor ( aka BAM ) instead.

In essence, one needs to toggle: -


from: -


to: -


in 101Custom.xml ( which I used to override the stock 100Custom.xml ) and then restarted Process Server.

Works for me, your mileage may vary.

Eric Herness, Chief Technology Officer for IBM BPM, has also blogged on the subject here: -

Wednesday, 17 September 2014

IBM Business Process Manager on Cloud service adds case handling and enhanced mobile UIs

New features built into IBM® Business Process Manager (IBM BPM) on Cloud include:

• Basic case-management capabilities that enable knowledge workers to drive business outcomes by using a combination of structured workflows, ad-hoc tasks, and document processing.
• New design capabilities for creating responsive user interfaces that can be designed once and run on any device form factor (phone, tablet, or desktop), to support mobile-ready process applications.
• Service availability for People's Republic of China with hosting in the IBM SoftLayer Hong Kong data center

IBM BPM on Cloud is a comprehensive and consumable, business process management (BPM) cloud service that delivers visibility and management of your business processes in a cloud environment. It includes tooling and runtime to design and run processes and provides capabilities for monitoring and optimizing work that is run within the platform. It is specifically designed to enable process owners and business users to get started with business process improvement quickly with a ready-to-use, cloud-based environment that is hosted in IBM cloud data centers and managed by IBM.