pgrep -lf jdk
# Starting time or date of a process
ps -o start_time <pid>
# Elapsed time of a process
ps -o "%t" <pid>

# Convert an RSA public key from PEM to DER
openssl rsa -inform PEM -in test-public-key.pem -outform DER -out test-public-key.der -pubin

# Convert a X509 certificate from PEM to DER
openssl x509 -in test-x509-cert.pem -inform PEM -out test-x509-cert.der -outform DER

# Print a X509 certificate
openssl x509 -text -in test-public-cert.pem

# Import a certificate to JKS
jdk1.6.0/bin/keytool -storepass changeit -keystore truststore.jks -importcert -alias mycert -file mycert.der -trustcacerts

# http://www.agentbob.info/agentbob/79.html
# Read the javadoc in ImportKey.java for details of converting it to a keystore

# Convert a private key from PEM to DER
openssl pkcs8 -topk8 -nocrypt -in private-key.pem -out private-key.der -outform der

# Convert cert from PEM to DER
openssl x509 -in cert.pem -out cert.der -outform der

# Convert private key and cert to keystore
java -cp . ImportKey private-key.der cert.der

# Change the storepass
keytool -keystore keystore.jks -storepass importkey -storepasswd -new changeit

# Change the alias
keytool -keystore keystore.jks -storepass changeit -changealias -alias importkey -keypass importkey -destalias myhost

# Change the keypass
keytool -keystore keystore.jks -storepass changeit -alias myhost -keypasswd -keypass importkey -new changeit

# List the keystore
keytool -keystore keystore.jks -storepass changeit -list -v

# How to generate a self-signed key pairs using keytool ?
keytool -genkeypair -dname "cn=myname,ou=myunit,o=myorg,c=AU" -alias myalias -keypass changeit -keystore ./my-keystore.jks
-keypass changeit -storepass changeit -validity 9999 -v

### Extracting the certificate (public key) ###

# Export the X509 certificate
keytool -export -alias myalias -keystore my-keystore.jks -storepass changeit -file mycert.der

# Display it
openssl x509 -noout -text -in mycert.der -inform der

# Convert to PEM
openssl x509 -out mycert.pem -outform pem -in mycert.der -inform der

### Extracting the private key ###

#Download, compile & run ExportPriv.

# Export private key into pcks8 format
javac ExportPriv.java
java ExportPriv my-keystore.jks myalias changeit > my-key.pkcs8

# Combine public and private key into pkcs12 format
openssl pkcs12 -export -out my-key.p12 -inkey my-key.pkcs8 -in my-cert.pem

# Convert pkc12 to PEM so it can be displayed
openssl pkcs12 -in pkcs-12-certificate(-and-key-file) -out pem-certificate(-and-key-file)

# Find out the MD5 of an X509 cert
openssl x509 -fingerprint -md5 -in cert.pem 

Recently I encountered a resource loading problem in JGroups from within a Tomcat webapp when trying to construct a JChannel when so triggered from a JMX MBean via the jconsole:

    channel = new JChannel("tcp.xml");

Basically if the JChannel is constructed from the webapp, it can locate the "tcp.xml" as a resource and works just fine. However if the JChannel is constructed from an MBean call stack via the jconsole, it failed to locate the resource due to the fact that the context classloading of the executing thread is different. So I hacked org.jgroups.util.Util to make it work in both cases. I made the hack on jgroups 2.2.9.4 , but I tried the latest 2.5.0CR1 which also seemed to exhibit the same problem/behavior. More details below.

My question is: should the hack I made be incorporated into the official version ? Guess what ? The next day, Bela Ban, the JGroups Lead / JBoss Clustering team, unconditionally accepted this change into the official JGroups 2.4 branch and the CVS head. Yeah!

public class Util {
...
    public static InputStream getResourceAsStream(String name, Class clazz) {
        ClassLoader loader;
                                                                                                                   
        try {
            loader=Thread.currentThread().getContextClassLoader();
            if(loader != null) {
// hack: returns only if resource is found
//                return loader.getResourceAsStream(name);
              InputStream is = loader.getResourceAsStream(name);
                                                                                                                   
              if (is != null)
                  return is;
            }
        }
        catch(Throwable t) {
        }
                                                                                                                   
        if(clazz != null) {
            try {
                loader=clazz.getClassLoader();
                if(loader != null) {
// hack: returns only if resource is found
//                    return loader.getResourceAsStream(name);
                    InputStream is = loader.getResourceAsStream(name);
                                                                                                                   
                    if (is != null)
                        return is;
                }
            }
            catch(Throwable t) {
            }
        }
        try {
            loader= ClassLoader.getSystemClassLoader();
            if(loader != null) {
                return loader.getResourceAsStream(name);
            }
        }
        catch(Throwable t) {
        }
                                                                                                                   
        return null;
    }
    ...
}

Continuing on the previous post on "Guaranteed db activation in all JVM's before table unlock in HA-JDBC ?"...

Actually, I think just doing a synchronous notification is not sufficient. What is required is an atomic/transactional activation of the specified db in all nodes - either all commit or rollback.

This naturally brings up the thought of using 2PC (Two Phase Commit) protocol. Here is a proposal. Say we have a synchronizing node S(ync), and two other members F(oo) and B(ar):

2PC Activation Protocol

(Assuming tables in the active db are locked, and synchronization has just completed and S is now trying to atomically/transactionally activate the inactive db in all JVM's before unlocking the tables.)

From S's perspective:

1. S synchronously sends an "ActivationReady" message to F and B. If all such sending failed, simply aborts this 2PC activation procedure (ie inactive db stays inactive). If some but not all such sending failed, S sends an "ActivationAbort" message to others that it has previously successfully sent the ActivationReady message, and then simply aborts this 2PC activation procedure (ie inactive db stays inactive). Else proceed to (2).


2. S synchronously sends an "ActivationCommit" message to F and B and then always activates the inactive db locally (regardless of whether any or all of the sending was successful.)
   
   (Note that S always unlocks the tables in the active db afterwards, regardless of whether the 2PC activation has been aborted or not. However, if S detects any possible failure during commit or rollback, it must wait for the other nodes to timeout before unlocking the tables.)

From F or B's perspective:

• If it receives an ActivationReady message followed by an ActivationAbort message, do nothing.
• If it receives an ActivationReady message followed by an ActivationCommit message, activate the specified db locally.
  
• In the extreme case that after an ActivationReady message is received but failed to received a subsequent ActivationCommit or ActivationAbort message received within, say, 5 secs, deactivate all db's locally. (This is to prevent any possible compromise to data consistency caused by not knowing whether the db in concern has been activated in other JVM's or not.)

Too complex ? Well, Paul Ferraro, the creator of HA-JDBC, seems to think so:

"I think you are over-engineering this a bit...

Database activation does not benefit from two-phase commit (2pc). The purpose of 2pc is to provide distributed consistency to a fallible operation. The operation in this case, activating a database, is not fallible - it is merely the insertion of a element into a collection, i.e. Balancer.add(...).

If you are curious about an instance where 2pc is warranted, check out the DistributableLockManager in HA-JDBC 2.0, which utilizes the JGroups TwoPhaseVotingAdapter."

But then what should we do in the case when the synchronous activation notification (presuming it has been implemented) to other JVM's resulted in partial success ? In other words, some get notified (and therefore have the inactive db activated) and some not ?

Wouldn't that result in data inconsistency (since some JVM which are succesfully notified would start to write to all the active db's in the cluster, whereas some JVM which are not succesfully notified would continue to write to only some but not all active db's in the cluster) ?

Some interesting comments from Bela Ban, the JGroups Lead/ JBoss Clustering team:

"Yes, if you want atomicity guarantees, you either have to use a uniform protocol (a message is only delivered if all receivers acked it, otherwise it will not get delivered at anyone) or 2PC as you outlined below. Uniform delivery is not part of JGroups, although it is on the todo list (http://jira.jboss.com/jira/browse/JGRP-138). I once had an impl, but it was too slow so I trashed it.

2PC is probably the way to go. I've done this in JBossCache: if you look at the code that is run when you have (a) a transaction and (b)synchronous method calls, then I pretty much run the classic PREPARE --> success? COMMIT : ROLLBACK pattern.

What you describe is such an implementation, but I guess that's something to be discussed in the scope of HA-JDBC. JGroups just provides the reliable transport with retransmission (lossless transport) and ordering."

More tips from Bela Ban:

"The RequestCorrelator is used by both MessageDispatcher and RpcDispatcher, it is at a lower abstraction level. I would suggest RpcDispatcher (for sync or async cluster-wide method calls) or MessageDispatcher (for sync or async cluster-wide message sending). Note that RpcDispatcher uses MessageDispatcher which uses RequestCorrelator.

In both RpcDispatcher.callRemoteMethods() and MessageDispatcher.castMessage() you get back an RspList which contains the result per member:

• The result (null if method was void)
• Whether the member was suspected, e.g. if you invoke foo() on {A,B,C,D}, and C crashes while you wait for acks, then C will be flagged as suspected in the response list
• Whether we received an ack (if the call is bounded by a timeout)

Paul, I'd suggest you use an {Rpc/Message}Dispatcher directly rather than NotificationBus. NB was designed to be a simple asynchronous notification mechanism."

What do you think ?

Say we have multiple JVM members accessing a HA-JDBC database cluster which involves a table locking synchronization strategy. Currently after a (inactive) db is synchronized, the code in ha-jdbc 1.1.11, method

    private void LocalDatabaseCluster#activate(Database, List, SynchronizationStrategy) 
        throws java.sql.SQLException

would activate the inactive db (both in the local JVM and other JVM's via jgroup notification) before executing a rollback which would unlock the tables in the active db.

My question is that when the JGroups notification of the activation to other JVM's is returned, does it mean
a) all other JVM's have successfully received the notification, or
b) some (or none or all) of the other JVM's have successfully received the notification, or
c) whether this only means the notification has been sent to other JVM's which may or may not receive the notification ?

If it is (b) or (c), is there any way to achieve/guarantee (a) ? The reason is that, for data consistency purposes, I am trying to provide the guarantee that all JVM's must have the (inactive) database activated before the table locks are released, or otherwise I'd rather to have the activation fail (ie inactive db stays inactive).

Unfortunately, at least for now, it turns out the answer is (c), as quoted from Paul Ferraro, the creator of HA-JDBC, at the ha-jdbc-user forum:

"Excellent question...

The distributed cluster state is implemented using a JGroups NotificationBus. Database activations and deactivations trigger appropriate notification events to listening members. I had always thought that NotificationBus operated synchronously, i.e. that NotificationBus.sendNotification(...) would not return until all nodes complete their NotificationBus.Consumer.handleNotificaiton(...) listener methods. I just completed a test against both JGroups 2.2.9.4 and 2.5-beta-2 - unfortunately, this is not the case. A synchronous messaging bus requires the use of a RequestCollator, and is not used by NotificationBus. I don't remember at what point I made this assumption - but I was wrong.

So the answer to your question - although my intention was (a), currently, it is (c). This needs to change. I need to re-implement DistributableDatabaseCluster (DistributableStateManager, in 2.0) to use a MessageDispatcher, DistributedQueue, or some other construct that utilizes a RequestCollator...."

Just realize JDI is such a powerful interface with implementation available in the jdk's tools.jar. Imagine you can write your own code to launch other VM's or attach to a running VM, inspect the executing threads, print out what methods the target VM is executing, etc. Essentially, with JDI, you can programmatically do whatever a debugger can with ease!

See here for some interesting sample code built on JDI.
See here for an interesting article by Gilad Bracha and David Ungar about Meta-Programming facilities somehow related to JDI.