Hmmm, security problems with WebSphere Application Server 6.1.0.29

I hit an interesting problem with my Lotus Connections 2.5 server today - the symptoms including my apparent inability to log in using the WAS administrator ID via the Integrated Solutions Console: -

https://lc25.uk.ibm.com:9043/ibm/console/logon.jsp

as well as these errors in the SystemOut log: -

[10/18/10 18:34:24:240 BST] 00000039 LTPAServerObj E   SECJ0369E: Authentication failed when using LTPA. The exception is <null>.
[10/18/10 18:36:32:799 BST] 0000003c FormLoginExte E   SECJ0118E: Authentication error during authentication for user lcadmin

and this error: -

ADMU0002E: Exception attempting to process server dmgr: javax.management.JMRuntimeException: ADMN0022E: Access is denied for the getState operation on Server MBean because of insufficient or empty credentials.

when I tried to administer the server from the command line using the command: -

/opt/IBM/WebSphere/AppServer/profiles/Dmgr01/bin/serverStatus.sh -all -user lcadmin -password wpsadm1n

My first thought was that LDAP was down, so I checked the VM in which LDAP ( IBM Tivoli Directory Server 6.2 ) and DB2 are running, but all appeared to be well. I could PING the LDAP server from the Connections box, and also TELNET to it on port 389, indicating that LDAP was ready and willing to receive.

I even stopped and started LDAP, but to no avail.

However, when I looked further into the logs, by tailing the LDAP error log: -

/home/dsrdbm01/idsslapd-dsrdbm01/logs/ibmslapd.log

I found: -

10/18/2010 09:09:40 PM GLPSRV200I Initializing primary database and its connections.
10/18/2010 09:09:41 PM GLPRDB001E Error code -1 from function:" SQLConnect " ldapdb2b .
10/18/2010 09:09:41 PM GLPSRV064E Failed to initialize be_config.
10/18/2010 09:09:41 PM GLPSRV040E Server starting in configuration only mode due to errors.
10/18/2010 09:09:41 PM GLPCOM024I The extended Operation plugin is successfully loaded from libloga.so.
10/18/2010 09:09:41 PM GLPCOM024I The extended Operation plugin is successfully loaded from libidsfget.so.
10/18/2010 09:09:41 PM GLPSRV180I Pass-through authentication is disabled.
10/18/2010 09:09:41 PM GLPCOM003I Non-SSL port initialized to 389.
10/18/2010 09:09:41 PM GLPSRV009I 6.2.0.0 server started.
10/18/2010 09:09:41 PM GLPSRV036E Errors were encountered while starting the server; started in configuration only mode.

and, when I looked again at the logs generated by my custom start LDAP script -

su db2inst1 -c '/home/db2inst1/sqllib/adm/db2start'
export DB2INSTANCE=dsrdbm01
/opt/ibm/ldap/V6.2/sbin/ibmslapd start

I saw the same errors: -

GLPSRV200I Initializing primary database and its connections.
GLPRDB001E Error code -1 from function:" SQLConnect " ldapdb2b .
GLPSRV064E Failed to initialize be_config.
GLPSRV040E Server starting in configuration only mode due to errors.

Thinking that this was looking like a DB2 "problem", I decided to attempt a connection to the database to find out for sure: -

su - dsrdbm01
db2 connect to LDAPDB2B

which returned: -

SQL8001N  An attempt to connect to the database failed due to a product licence problem.  SQLSTATE=42968

which, according to: -

db2 ? SQL8001N

means: -

SQL8001N An attempt to connect to the database failed due to a product licence problem. 

Explanation:

The connection attempt failed due to one of the following DB2 licensing issues:

o   A license key is not present. 
o   A license key has not been correctly installed. 
o   A license key has expired. 
o   This DB2 product edition does not support a license key. 

User Response:

Instruct your database administrator to verify that a valid license key for the DB2 product is installed correctly, and has not expired. 

o   Use the db2licm -l command or DB2 License Center to view licenses.
o   Use the db2licm -a command or DB2 License Center to register a license.
o   You can obtain a license key for the product by contacting your IBM representative or authorized dealer. 

sqlcode :  -8001
sqlstate :  42968

Following the recommendation, I checked the DB2 license key: -

db2licm -l

which returned: -

Product name:                     "DB2 Enterprise Server Edition"
Expiry date:                      "Expired"
Product identifier:               "db2ese"
Version information:              "9.1"

Thankfully, I had the DB2 Enterprise Server Edition 9.1 license key available, in a previously downloaded package on my USB drive.

I installed it using the command: -

db2licm -a ~/db2ese_t.lic

and re-ran the command: -

db2licm -l

which now returns: -

Product name:                     "DB2 Enterprise Server Edition"
License type:                     "Trial"
Expiry date:                      "01/15/2011"
Product identifier:               "db2ese"
Version information:              "9.1"
Annotation:                       "-5;(_t)"

and also re-tested the connection: -

db2 connect to LDAPDB2B

which now returns: -

   Database Connection Information

Database server        = DB2/LINUX 9.1.9
SQL authorization ID   = DSRDBM01
Local database alias   = LDAPDB2B

Finally, I stopped LDAP: -

export DB2INSTANCE=dsrdbm01
/opt/ibm/ldap/V6.2/sbin/ibmslapd -k

and started it again: -

su db2inst1 -c '/home/db2inst1/sqllib/adm/db2start'
export DB2INSTANCE=dsrdbm01
/opt/ibm/ldap/V6.2/sbin/ibmslapd start

Once I restarted my WAS instance ( sadly I had to force it to quit using the "nuclear" kill switch of killall java ), I was able to authenticate into WAS using the ISC, run commands such as serverStatus.sh etc.

The moral of the story - if you get an authentication problem with WAS, check that your DB2 license hasn't expired :-)