Revisited: PGP Encryption/Decryption in Java

One of the most popular posts on my blog is my article about PGP Encryption and Decryption. I’ve had a lot of great questions as well as responses from user who have even been kind enough to post modifications to the original code to allow it to do more.

I thought I would revisit the post to provide a new version of the code that incorporated what some readers (Mateusz Klos, Juraj, and Sumit in particular) provided.

At the same time I thought I would upgrade the code to the latest library from Bouncy Castle. Yeah, so, boy did they change some things. The latest examples from Bouncy Castle use code that they have deprecated (they didn’t update their examples). There are no examples on how to use the new API’s or how to correct the deprecations. The API does mention (“use this instead”) which is great and more than most projects provide however it was only a part of the solution.

Anyway, after some time I’ve gotten the code to use the latest Bouncy Castle libraries (1.47).

The Code

First is the Util which does the real work. It has been upgraded to 1.47 and contains suggestions from readers. I should point out that two methods are from LockBox Lobs PGP Encryption tools. I didn’t see the point of requiring a rather LARGE library for two little methods so I just put them in here.


import java.io.ByteArrayOutputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.security.NoSuchProviderException;
import java.security.Provider;
import java.security.SecureRandom;
import java.security.Security;
import java.util.Date;
import java.util.Iterator;

import org.apache.commons.io.IOUtils;
import org.bouncycastle.bcpg.ArmoredOutputStream;
import org.bouncycastle.bcpg.HashAlgorithmTags;
import org.bouncycastle.bcpg.PublicKeyAlgorithmTags;
import org.bouncycastle.bcpg.sig.KeyFlags;
import org.bouncycastle.jce.provider.BouncyCastleProvider;
import org.bouncycastle.openpgp.PGPCompressedData;
import org.bouncycastle.openpgp.PGPCompressedDataGenerator;
import org.bouncycastle.openpgp.PGPEncryptedData;
import org.bouncycastle.openpgp.PGPEncryptedDataGenerator;
import org.bouncycastle.openpgp.PGPEncryptedDataList;
import org.bouncycastle.openpgp.PGPException;
import org.bouncycastle.openpgp.PGPLiteralData;
import org.bouncycastle.openpgp.PGPLiteralDataGenerator;
import org.bouncycastle.openpgp.PGPObjectFactory;
import org.bouncycastle.openpgp.PGPOnePassSignature;
import org.bouncycastle.openpgp.PGPOnePassSignatureList;
import org.bouncycastle.openpgp.PGPPrivateKey;
import org.bouncycastle.openpgp.PGPPublicKey;
import org.bouncycastle.openpgp.PGPPublicKeyEncryptedData;
import org.bouncycastle.openpgp.PGPPublicKeyRing;
import org.bouncycastle.openpgp.PGPPublicKeyRingCollection;
import org.bouncycastle.openpgp.PGPSecretKey;
import org.bouncycastle.openpgp.PGPSecretKeyRing;
import org.bouncycastle.openpgp.PGPSecretKeyRingCollection;
import org.bouncycastle.openpgp.PGPSignature;
import org.bouncycastle.openpgp.PGPSignatureGenerator;
import org.bouncycastle.openpgp.PGPSignatureList;
import org.bouncycastle.openpgp.PGPSignatureSubpacketGenerator;
import org.bouncycastle.openpgp.PGPSignatureSubpacketVector;
import org.bouncycastle.openpgp.PGPUtil;
import org.bouncycastle.openpgp.operator.PBESecretKeyDecryptor;
import org.bouncycastle.openpgp.operator.PGPContentSignerBuilder;
import org.bouncycastle.openpgp.operator.bc.BcPBESecretKeyDecryptorBuilder;
import org.bouncycastle.openpgp.operator.bc.BcPGPContentSignerBuilder;
import org.bouncycastle.openpgp.operator.bc.BcPGPContentVerifierBuilderProvider;
import org.bouncycastle.openpgp.operator.bc.BcPGPDataEncryptorBuilder;
import org.bouncycastle.openpgp.operator.bc.BcPGPDigestCalculatorProvider;
import org.bouncycastle.openpgp.operator.bc.BcPublicKeyDataDecryptorFactory;
import org.bouncycastle.openpgp.operator.bc.BcPublicKeyKeyEncryptionMethodGenerator;

public class PGPUtils {

    private static final int   BUFFER_SIZE = 1 << 16; // should always be power of 2
    private static final int   KEY_FLAGS = 27;
    private static final int[] MASTER_KEY_CERTIFICATION_TYPES = new int[]{
    	PGPSignature.POSITIVE_CERTIFICATION,
    	PGPSignature.CASUAL_CERTIFICATION,
    	PGPSignature.NO_CERTIFICATION,
    	PGPSignature.DEFAULT_CERTIFICATION
    };

    @SuppressWarnings("unchecked")
    public static PGPPublicKey readPublicKey(InputStream in)
    	throws IOException, PGPException
    {

        PGPPublicKeyRingCollection keyRingCollection = new PGPPublicKeyRingCollection(PGPUtil.getDecoderStream(in));

        //
        // we just loop through the collection till we find a key suitable for encryption, in the real
        // world you would probably want to be a bit smarter about this.
        //
        PGPPublicKey publicKey = null;

        //
        // iterate through the key rings.
        //
        Iterator<PGPPublicKeyRing> rIt = keyRingCollection.getKeyRings();

        while (publicKey == null && rIt.hasNext()) {
            PGPPublicKeyRing kRing = rIt.next();
            Iterator<PGPPublicKey> kIt = kRing.getPublicKeys();
            while (publicKey == null && kIt.hasNext()) {
                PGPPublicKey key = kIt.next();
                if (key.isEncryptionKey()) {
                    publicKey = key;
                }
            }
        }

        if (publicKey == null) {
            throw new IllegalArgumentException("Can't find public key in the key ring.");
        }
        if (!isForEncryption(publicKey)) {
            throw new IllegalArgumentException("KeyID " + publicKey.getKeyID() + " not flagged for encryption.");
        }

        return publicKey;
    }

    @SuppressWarnings("unchecked")
	public static PGPSecretKey readSecretKey(InputStream in)
		throws IOException, PGPException
	{

        PGPSecretKeyRingCollection keyRingCollection = new PGPSecretKeyRingCollection(PGPUtil.getDecoderStream(in));

        //
        // We just loop through the collection till we find a key suitable for signing.
        // In the real world you would probably want to be a bit smarter about this.
        //
        PGPSecretKey secretKey = null;

        Iterator<PGPSecretKeyRing> rIt = keyRingCollection.getKeyRings();
        while (secretKey == null && rIt.hasNext()) {
            PGPSecretKeyRing keyRing = rIt.next();
            Iterator<PGPSecretKey> kIt = keyRing.getSecretKeys();
            while (secretKey == null && kIt.hasNext()) {
                PGPSecretKey key = kIt.next();
                if (key.isSigningKey()) {
                    secretKey = key;
                }
            }
        }

        // Validate secret key
        if (secretKey == null) {
            throw new IllegalArgumentException("Can't find private key in the key ring.");
        }
        if (!secretKey.isSigningKey()) {
            throw new IllegalArgumentException("Private key does not allow signing.");
        }
        if (secretKey.getPublicKey().isRevoked()) {
            throw new IllegalArgumentException("Private key has been revoked.");
        }
        if (!hasKeyFlags(secretKey.getPublicKey(), KeyFlags.SIGN_DATA)) {
            throw new IllegalArgumentException("Key cannot be used for signing.");
        }

        return secretKey;
    }

    /**
     * Load a secret key ring collection from keyIn and find the private key corresponding to
     * keyID if it exists.
     *
     * @param keyIn input stream representing a key ring collection.
     * @param keyID keyID we want.
     * @param pass passphrase to decrypt secret key with.
     * @return
     * @throws IOException
     * @throws PGPException
     * @throws NoSuchProviderException
     */
    public  static PGPPrivateKey findPrivateKey(InputStream keyIn, long keyID, char[] pass)
    	throws IOException, PGPException, NoSuchProviderException
    {
        PGPSecretKeyRingCollection pgpSec = new PGPSecretKeyRingCollection(PGPUtil.getDecoderStream(keyIn));
        return findPrivateKey(pgpSec.getSecretKey(keyID), pass);

    }

    /**
     * Load a secret key and find the private key in it
     * @param pgpSecKey The secret key
     * @param pass passphrase to decrypt secret key with
     * @return
     * @throws PGPException
     */
    public static PGPPrivateKey findPrivateKey(PGPSecretKey pgpSecKey, char[] pass)
    	throws PGPException
    {
    	if (pgpSecKey == null) return null;

        PBESecretKeyDecryptor decryptor = new BcPBESecretKeyDecryptorBuilder(new BcPGPDigestCalculatorProvider()).build(pass);
        return pgpSecKey.extractPrivateKey(decryptor);
    }

    /**
     * decrypt the passed in message stream
     */
    @SuppressWarnings("unchecked")
	public static void decryptFile(InputStream in, OutputStream out, InputStream keyIn, char[] passwd)
    	throws Exception
    {
    	Security.addProvider(new BouncyCastleProvider());

        in = org.bouncycastle.openpgp.PGPUtil.getDecoderStream(in);

        PGPObjectFactory pgpF = new PGPObjectFactory(in);
        PGPEncryptedDataList enc;

        Object o = pgpF.nextObject();
        //
        // the first object might be a PGP marker packet.
        //
        if (o instanceof  PGPEncryptedDataList) {
            enc = (PGPEncryptedDataList) o;
        } else {
            enc = (PGPEncryptedDataList) pgpF.nextObject();
        }

        //
        // find the secret key
        //
        Iterator<PGPPublicKeyEncryptedData> it = enc.getEncryptedDataObjects();
        PGPPrivateKey sKey = null;
        PGPPublicKeyEncryptedData pbe = null;

        while (sKey == null && it.hasNext()) {
            pbe = it.next();

            sKey = findPrivateKey(keyIn, pbe.getKeyID(), passwd);
        }

        if (sKey == null) {
            throw new IllegalArgumentException("Secret key for message not found.");
        }

        InputStream clear = pbe.getDataStream(new BcPublicKeyDataDecryptorFactory(sKey));

        PGPObjectFactory plainFact = new PGPObjectFactory(clear);

        Object message = plainFact.nextObject();

        if (message instanceof  PGPCompressedData) {
            PGPCompressedData cData = (PGPCompressedData) message;
            PGPObjectFactory pgpFact = new PGPObjectFactory(cData.getDataStream());

            message = pgpFact.nextObject();
        }

        if (message instanceof  PGPLiteralData) {
            PGPLiteralData ld = (PGPLiteralData) message;

            InputStream unc = ld.getInputStream();
            int ch;

            while ((ch = unc.read()) >= 0) {
                out.write(ch);
            }
        } else if (message instanceof  PGPOnePassSignatureList) {
            throw new PGPException("Encrypted message contains a signed message - not literal data.");
        } else {
            throw new PGPException("Message is not a simple encrypted file - type unknown.");
        }

        if (pbe.isIntegrityProtected()) {
            if (!pbe.verify()) {
            	throw new PGPException("Message failed integrity check");
            }
        }
    }

    public static void encryptFile(
        OutputStream out,
        String fileName,
        PGPPublicKey encKey,
        boolean armor,
        boolean withIntegrityCheck)
        throws IOException, NoSuchProviderException, PGPException
    {
    	Security.addProvider(new BouncyCastleProvider());

        if (armor) {
            out = new ArmoredOutputStream(out);
        }

        ByteArrayOutputStream bOut = new ByteArrayOutputStream();
        PGPCompressedDataGenerator comData = new PGPCompressedDataGenerator(PGPCompressedData.ZIP);

        PGPUtil.writeFileToLiteralData(
                comData.open(bOut),
                PGPLiteralData.BINARY,
                new File(fileName) );

        comData.close();

        BcPGPDataEncryptorBuilder dataEncryptor = new BcPGPDataEncryptorBuilder(PGPEncryptedData.TRIPLE_DES);
        dataEncryptor.setWithIntegrityPacket(withIntegrityCheck);
        dataEncryptor.setSecureRandom(new SecureRandom());

        PGPEncryptedDataGenerator encryptedDataGenerator = new PGPEncryptedDataGenerator(dataEncryptor);
        encryptedDataGenerator.addMethod(new BcPublicKeyKeyEncryptionMethodGenerator(encKey));

        byte[] bytes = bOut.toByteArray();
        OutputStream cOut = encryptedDataGenerator.open(out, bytes.length);
        cOut.write(bytes);
        cOut.close();
        out.close();
    }

    @SuppressWarnings("unchecked")
	public static void signEncryptFile(
        OutputStream out,
        String fileName,
        PGPPublicKey publicKey,
        PGPSecretKey secretKey,
        String password,
        boolean armor,
        boolean withIntegrityCheck )
        throws Exception
    {

        // Initialize Bouncy Castle security provider
        Provider provider = new BouncyCastleProvider();
        Security.addProvider(provider);

        if (armor) {
            out = new ArmoredOutputStream(out);
        }

        BcPGPDataEncryptorBuilder dataEncryptor = new BcPGPDataEncryptorBuilder(PGPEncryptedData.TRIPLE_DES);
        dataEncryptor.setWithIntegrityPacket(withIntegrityCheck);
        dataEncryptor.setSecureRandom(new SecureRandom());

        PGPEncryptedDataGenerator encryptedDataGenerator = new PGPEncryptedDataGenerator(dataEncryptor);
        encryptedDataGenerator.addMethod(new BcPublicKeyKeyEncryptionMethodGenerator(publicKey));

        OutputStream encryptedOut = encryptedDataGenerator.open(out, new byte[PGPUtils.BUFFER_SIZE]);

        // Initialize compressed data generator
        PGPCompressedDataGenerator compressedDataGenerator = new PGPCompressedDataGenerator(PGPCompressedData.ZIP);
        OutputStream compressedOut = compressedDataGenerator.open(encryptedOut, new byte [PGPUtils.BUFFER_SIZE]);

        // Initialize signature generator
        PGPPrivateKey privateKey = findPrivateKey(secretKey, password.toCharArray());

        PGPContentSignerBuilder signerBuilder = new BcPGPContentSignerBuilder(secretKey.getPublicKey().getAlgorithm(),
                HashAlgorithmTags.SHA1);

        PGPSignatureGenerator signatureGenerator = new PGPSignatureGenerator(signerBuilder);
        signatureGenerator.init(PGPSignature.BINARY_DOCUMENT, privateKey);

        boolean firstTime = true;
        Iterator<String> it = secretKey.getPublicKey().getUserIDs();
        while (it.hasNext() && firstTime) {
            PGPSignatureSubpacketGenerator spGen = new PGPSignatureSubpacketGenerator();
            spGen.setSignerUserID(false, it.next());
            signatureGenerator.setHashedSubpackets(spGen.generate());
            // Exit the loop after the first iteration
            firstTime = false;
        }
        signatureGenerator.generateOnePassVersion(false).encode(compressedOut);

        // Initialize literal data generator
        PGPLiteralDataGenerator literalDataGenerator = new PGPLiteralDataGenerator();
        OutputStream literalOut = literalDataGenerator.open(
            compressedOut,
            PGPLiteralData.BINARY,
            fileName,
            new Date(),
            new byte [PGPUtils.BUFFER_SIZE] );

        // Main loop - read the "in" stream, compress, encrypt and write to the "out" stream
        FileInputStream in = new FileInputStream(fileName);
        byte[] buf = new byte[PGPUtils.BUFFER_SIZE];
        int len;
        while ((len = in.read(buf)) > 0) {
            literalOut.write(buf, 0, len);
            signatureGenerator.update(buf, 0, len);
        }

        in.close();
        literalDataGenerator.close();
        // Generate the signature, compress, encrypt and write to the "out" stream
        signatureGenerator.generate().encode(compressedOut);
        compressedDataGenerator.close();
        encryptedDataGenerator.close();
        if (armor) {
            out.close();
        }
    }

    public static boolean verifyFile(
    	InputStream in,
        InputStream keyIn,
        String extractContentFile)
        throws Exception
    {
        in = PGPUtil.getDecoderStream(in);

        PGPObjectFactory pgpFact = new PGPObjectFactory(in);
        PGPCompressedData c1 = (PGPCompressedData)pgpFact.nextObject();

        pgpFact = new PGPObjectFactory(c1.getDataStream());

        PGPOnePassSignatureList p1 = (PGPOnePassSignatureList)pgpFact.nextObject();

        PGPOnePassSignature ops = p1.get(0);

        PGPLiteralData p2 = (PGPLiteralData)pgpFact.nextObject();

        InputStream dIn = p2.getInputStream();

        IOUtils.copy(dIn, new FileOutputStream(extractContentFile));

        int ch;
        PGPPublicKeyRingCollection pgpRing = new PGPPublicKeyRingCollection(PGPUtil.getDecoderStream(keyIn));

        PGPPublicKey key = pgpRing.getPublicKey(ops.getKeyID());

        FileOutputStream out = new FileOutputStream(p2.getFileName());

        ops.init(new BcPGPContentVerifierBuilderProvider(), key);

        while ((ch = dIn.read()) >= 0)
        {
            ops.update((byte)ch);
            out.write(ch);
        }

        out.close();

        PGPSignatureList p3 = (PGPSignatureList)pgpFact.nextObject();
        return ops.verify(p3.get(0));
    }

    /**
     * From LockBox Lobs PGP Encryption tools.
     * http://www.lockboxlabs.org/content/downloads
     *
     * I didn't think it was worth having to import a 4meg lib for three methods
     * @param key
     * @return
     */
    public static boolean isForEncryption(PGPPublicKey key)
    {
        if (key.getAlgorithm() == PublicKeyAlgorithmTags.RSA_SIGN
            || key.getAlgorithm() == PublicKeyAlgorithmTags.DSA
            || key.getAlgorithm() == PublicKeyAlgorithmTags.EC
            || key.getAlgorithm() == PublicKeyAlgorithmTags.ECDSA)
        {
            return false;
        }

        return hasKeyFlags(key, KeyFlags.ENCRYPT_COMMS | KeyFlags.ENCRYPT_STORAGE);
    }

    /**
     * From LockBox Lobs PGP Encryption tools.
     * http://www.lockboxlabs.org/content/downloads
     *
     * I didn't think it was worth having to import a 4meg lib for three methods
     * @param key
     * @return
     */
    @SuppressWarnings("unchecked")
	private static boolean hasKeyFlags(PGPPublicKey encKey, int keyUsage) {
        if (encKey.isMasterKey()) {
            for (int i = 0; i != PGPUtils.MASTER_KEY_CERTIFICATION_TYPES.length; i++) {
                for (Iterator<PGPSignature> eIt = encKey.getSignaturesOfType(PGPUtils.MASTER_KEY_CERTIFICATION_TYPES[i]); eIt.hasNext();) {
                    PGPSignature sig = eIt.next();
                    if (!isMatchingUsage(sig, keyUsage)) {
                        return false;
                    }
                }
            }
        }
        else {
            for (Iterator<PGPSignature> eIt = encKey.getSignaturesOfType(PGPSignature.SUBKEY_BINDING); eIt.hasNext();) {
                PGPSignature sig = eIt.next();
                if (!isMatchingUsage(sig, keyUsage)) {
                    return false;
                }
            }
        }
        return true;
    }

    /**
     * From LockBox Lobs PGP Encryption tools.
     * http://www.lockboxlabs.org/content/downloads
     *
     * I didn't think it was worth having to import a 4meg lib for three methods
     * @param key
     * @return
     */
    private static boolean isMatchingUsage(PGPSignature sig, int keyUsage) {
        if (sig.hasSubpackets()) {
            PGPSignatureSubpacketVector sv = sig.getHashedSubPackets();
            if (sv.hasSubpacket(PGPUtils.KEY_FLAGS)) {
                // code fix suggested by kzt (see comments)
                if ((sv.getKeyFlags() == 0 && keyUsage) == 0) {
                    return false;
                }
            }
        }
        return true;
    }

}

The PGPFileProcessor is a bean based implementation of calling the Util so that you can easily integrate with Spring.


import java.io.FileInputStream;
import java.io.FileOutputStream;

import org.bouncycastle.openpgp.PGPPublicKey;
import org.bouncycastle.openpgp.PGPSecretKey;

public class PGPFileProcessor {

    private String passphrase;
    private String publicKeyFileName;
    private String secretKeyFileName;
    private String inputFileName;
    private String outputFileName;
    private boolean asciiArmored = false;
    private boolean integrityCheck = true;

    public boolean encrypt() throws Exception {
        FileInputStream keyIn = new FileInputStream(publicKeyFileName);
        FileOutputStream out = new FileOutputStream(outputFileName);
        PGPUtils.encryptFile(out, inputFileName, PGPUtils.readPublicKey(keyIn), asciiArmored, integrityCheck);
        out.close();
        keyIn.close();
        return true;
    }

    public boolean signEncrypt() throws Exception {
        FileOutputStream out = new FileOutputStream(outputFileName);
        FileInputStream publicKeyIn = new FileInputStream(publicKeyFileName);
        FileInputStream secretKeyIn = new FileInputStream(secretKeyFileName);

        PGPPublicKey publicKey = PGPUtils.readPublicKey(publicKeyIn);
        PGPSecretKey secretKey = PGPUtils.readSecretKey(secretKeyIn);

        PGPUtils.signEncryptFile(
                out,
                this.getInputFileName(),
                publicKey,
                secretKey,
                this.getPassphrase(),
                this.isAsciiArmored(),
                this.isIntegrityCheck() );

        out.close();
        publicKeyIn.close();
        secretKeyIn.close();

        return true;
    }

    public boolean decrypt() throws Exception {
        FileInputStream in = new FileInputStream(inputFileName);
        FileInputStream keyIn = new FileInputStream(secretKeyFileName);
        FileOutputStream out = new FileOutputStream(outputFileName);
        PGPUtils.decryptFile(in, out, keyIn, passphrase.toCharArray());
        in.close();
        out.close();
        keyIn.close();
        return true;
    }

    public boolean isAsciiArmored() {
            return asciiArmored;
    }

    public void setAsciiArmored(boolean asciiArmored) {
            this.asciiArmored = asciiArmored;
    }

    public boolean isIntegrityCheck() {
            return integrityCheck;
    }

    public void setIntegrityCheck(boolean integrityCheck) {
            this.integrityCheck = integrityCheck;
    }

    public String getPassphrase() {
            return passphrase;
    }

    public void setPassphrase(String passphrase) {
            this.passphrase = passphrase;
    }

    public String getPublicKeyFileName() {
            return publicKeyFileName;
    }

    public void setPublicKeyFileName(String publicKeyFileName) {
            this.publicKeyFileName = publicKeyFileName;
    }

    public String getSecretKeyFileName() {
            return secretKeyFileName;
    }

    public void setSecretKeyFileName(String secretKeyFileName) {
            this.secretKeyFileName = secretKeyFileName;
    }

    public String getInputFileName() {
            return inputFileName;
    }

    public void setInputFileName(String inputFileName) {
            this.inputFileName = inputFileName;
    }

    public String getOutputFileName() {
            return outputFileName;
    }

    public void setOutputFileName(String outputFileName) {
            this.outputFileName = outputFileName;
    }

}

Examples

The code is still set up in a bean standard manner to support easy integration with Spring. The examples are very simple but should give more than enough guidance for basic usage. If there is any confusion please ask.

public class Tester {

	private static final String PASSPHRASE = "test";

	private static final String DE_INPUT = "src/test/x.pgp";
	private static final String DE_OUTPUT = "src/test/x.txt";
	private static final String DE_KEY_FILE = "src/test/secring.skr";

	private static final String E_INPUT = "src/test/x.txt";
	private static final String E_OUTPUT = "src/test/x.pgp";
	private static final String E_KEY_FILE = "src/test/pubring.pkr";


	public static void testDecrypt() throws Exception {
		PGPFileProcessor p = new PGPFileProcessor();
		p.setInputFileName(DE_INPUT);
		p.setOutputFileName(DE_OUTPUT);
		p.setPassphrase(PASSPHRASE);
		p.setSecretKeyFileName(DE_KEY_FILE);
		System.out.println(p.decrypt());
	}

	public static void testEncrypt() throws Exception {
		PGPFileProcessor p = new PGPFileProcessor();
		p.setInputFileName(E_INPUT);
		p.setOutputFileName(E_OUTPUT);
		p.setPassphrase(PASSPHRASE);
		p.setPublicKeyFileName(E_KEY_FILE);
		System.out.println(p.encrypt());
	}
}

Conclusion

That’s it. I hope you don’t feel I’m cheating out of a post for this month but I thought this would be very helpful since so many people read the original post.

I should also warn that this code is not in my production env at work because we haven’t upgraded our encryption libraries yet. This means there could be a bug lurking in there somewhere. If you find one please tell me and I’ll correct it.

Question: (Lets see if anyone reads down this far): Would you prefer that I bundle the code into a little util jar to d/l to make it easier to use?

Posted in Java, Programming | Tagged , , | 126 Comments

Spring Custom Tags (Extensible XML) – Part 2

The DefinitionParser

Ok, we’ve got it all set up, now we need to code the thing. Let me state, if you didn’t grasp this, that this is more of an advanced example. If you want a basic example take a look at the Spring documentation linked to at the top of the document.

You still here? Ok, I warned you. Follow me into the depths of code 🙂

Well, what does custom tag code do? In a nutshell it is in charge of registering a org.springframework.beans.factory.FactoryBean wherever your tag exists in your XML so that when it comes time to use the value that your tag represents it can easily get it following the Spring FactoryBean methodology.

Any class that wishes to handle the parsing of custom tags must implement org.springframework.beans.factory.xml.BeanDefinitionParser. It’s just an interface that passed in the W3C Dom Element that is the tag itself and the Context in which the parsing is occurring. The Element object is easy enough, but what is the Context?

Spring passes you an instance of org.springframework.beans.factory.xml.ParserContext that represents where the tag is in the context of things (example: is it a nested tag?) as well as references to the BeanDefinitionRegistry (where Spring keeps info on all the beans you have every defined) and a few other things. This is helpful later as you may want to lookup another bean or perhaps treat the bean created from your tag differently if it is nested.

Lets approach this by following the tag structure. First we will write the code that will parse the fileList followed by the fileFilter. Lastly we will code a util to handle the Spring bean/ref/idref/value tags. I’m making this a util since it might be handy for other tags in the future.

FileListDefinitionParser

The FileListDefinitionParser is in charge of handling the parsing of the fileList tag. To make this easier to follow I’m going to put the comments about the code directly in the code (my college professors would be so proud of me) so that I can explain things better.

package com.example.core.commons.tag;

import java.io.File;
import java.io.FileFilter;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
import java.util.HashSet;
import java.util.List;

import org.springframework.beans.factory.FactoryBean;
import org.springframework.beans.factory.support.BeanDefinitionBuilder;
import org.springframework.beans.factory.support.ManagedList;
import org.springframework.beans.factory.xml.AbstractSingleBeanDefinitionParser;
import org.springframework.beans.factory.xml.ParserContext;
import org.springframework.util.xml.DomUtils;
import org.w3c.dom.Element;
import org.w3c.dom.NodeList;

/**
 * Returns a list of files that are in a directory.  The list may be limited
 * by nesting a core-commons:fileFilter tag that will do the intersection
 * of all the fileFilters vs what is in the directory.
 * <p/>
 *
 * Also supports the nesting of beans, idref, ref, and value tags that return
 * File objects (the value tag's value will be converted to a new File)
 *
 * <p/>
 * Examples:
 * <br/>
 *
 * Get all the files in the current directory
 * <core-commons:fileList directory="."/>
 *
 * Get all the files in the current directory that end in XML
 * <core-commons:fileList directory=".">
 * 		<core-commons:fileFilter>
 *      	<bean class="org.apache.commons.io.filefilter.RegexFileFilter">
 *          	<constructor-arg value=".*.xml"/>
 *           </bean>
 *      </core-commons:fileFilter>
 * </core-commons:fileList>
 *
 * Get all the files in the current directory that end in XML (specify the fileList separately)
 * <core-commons:fileList>
 *  	<core-commons:fileFilter>
 *      	<bean class="org.apache.commons.io.filefilter.RegexFileFilter">
 *          	<constructor-arg value=".*.xml"/>
 *          </bean>
 *      </core-commons:fileFilter>
 *      <core-commons:fileList directory="."/>
 * </core-commons:fileList>
 *
 * Get all files in the /tmp and /something directory that end in .xml
 * <core-commons:fileList directory="/tmp">
 * 		<core-commons:fileFilter>
 * 			<bean class="org.apache.commons.io.filefilter.RegexFileFilter">
 * 				<constructor-arg value=".*.xml"/>
 * 			</bean>
 * 		</core-commons:fileFilter>
 * 		<core-commons:fileList directory="/something"/>
 * </core-commons:fileList>
 *
 * Get all files in the /tmp and /something directory that end in .xml and can be written to
 * <core-commons:fileList directory="/tmp">
 * 		<core-commons:fileFilter>
 * 			<bean class="org.apache.commons.io.filefilter.CanWriteFileFilter"/>
 * 			<bean class="org.apache.commons.io.filefilter.RegexFileFilter">
 * 				<constructor-arg value=".*.xml"/>
 * 			</bean>
 * 		</core-commons:fileFilter>
 * 		<core-commons:fileList directory="/something"/>
 * </core-commons:fileList>
 *
 * Get all files in the /tmp directory and the pom.xml file in another directory
 * <core-commons:fileList>
 * 		<core-commons:fileList directory="/tmp"/>
 * 		<value>pom.xml</value>
 * </core-commons:fileList>
 *
 * @author seamans
 *
 */
public class FileListDefinitionParser
	extends AbstractSingleBeanDefinitionParser
{

	/**
	 * The bean that is created for this tag element
	 * 
	 * @param element The tag element
	 * @return A FileListFactoryBean
	 */
	@Override
	protected Class<?> getBeanClass(Element element) {
		return FileListFactoryBean.class;
	}

	/**
	 * Called when the fileList tag is to be parsed
	 * 
	 * @param element The tag element
	 * @param ctx The context in which the parsing is occuring
	 * @param builder The bean definitions build to use
	 */
	@Override
	protected void doParse(Element element, ParserContext ctx, BeanDefinitionBuilder builder) {
		// Set the directory property
		builder.addPropertyValue("directory", element.getAttribute("directory"));
		
		// Set the scope
		builder.setScope(element.getAttribute("scope"));

		// We want any parsing to occur as a child of this tag so we need to make
		// a new one that has this as it's owner/parent
		ParserContext nestedCtx = new ParserContext(ctx.getReaderContext(), ctx.getDelegate(), builder.getBeanDefinition());

		// Support for filters
		Element exclusionElem = DomUtils.getChildElementByTagName(element, "fileFilter");
		if (exclusionElem != null) {
			// Just make a new Parser for each one and let the parser do the work
			FileFilterDefinitionParser ff = new FileFilterDefinitionParser();
			builder.addPropertyValue("filters", ff.parse(exclusionElem, nestedCtx));
		}

		// Support for nested fileList
		List<Element> fileLists = DomUtils.getChildElementsByTagName(element, "fileList");
		// Any objects that created will be placed in a ManagedList
		// so Spring does the bulk of the resolution work for us
		ManagedList<Object> nestedFiles = new ManagedList<Object>();
		if (fileLists.size() > 0) {
			// Just make a new Parser for each one and let them do the work
			FileListDefinitionParser fldp = new FileListDefinitionParser();
			for (Element fileListElem : fileLists) {
				nestedFiles.add(fldp.parse(fileListElem, nestedCtx));
			}
		}

		// Support for other tags that return File (value will be converted to file)
		try {
			// Go through any other tags we may find.  This does not mean we support
			// any tag, we support only what parseLimitedList will process
			NodeList nl = element.getChildNodes();
			for (int i=0; i<nl.getLength(); i++) {
				// Parse each child tag we find in the correct scope but we 
				// won't support custom tags at this point as it coudl destablize things
				DefinitionParserUtil.parseLimitedList(nestedFiles, nl.item(i), ctx,
					builder.getBeanDefinition(), element.getAttribute("scope"), false);
			}
		}
		catch (Exception e) {
			throw new RuntimeException(e);
		}

		// Set the nestedFiles in the properties so it is set on the FactoryBean
		builder.addPropertyValue("nestedFiles", nestedFiles);

	}

	public static class FileListFactoryBean
		implements FactoryBean<Collection<File>>
	{

		String directory;
		private Collection<FileFilter> filters;
		private Collection<File> nestedFiles;

		@Override
		public Collection<File> getObject() throws Exception {
			// These can be an array list because the directory will have unique's and the nested is already only unique's
			Collection<File> files = new ArrayList<File>();
			Collection<File> results = new ArrayList<File>(0);

			if (directory != null) {
				// get all the files in the directory
				File dir = new File(directory);
				File[] dirFiles = dir.listFiles();
				if (dirFiles != null) {
					files = Arrays.asList(dirFiles);
				}
			}

			// If there are any files that were created from the nested tags,
			// add those to the list of files
			if (nestedFiles != null) {
				files.addAll(nestedFiles);
			}

			// If there are filters we need to go through each filter
			// and see if the files in the list pass the filters.
			// If the files does not pass any one of the filters then it
			// will not be included in the list
			if (filters != null) {
				boolean add;
				for (File f : files) {
					add = true;
					for (FileFilter ff : filters) {
						if (!ff.accept(f)) {
							add = false;
							break;
						}
					}
					if (add) results.add(f);
				}
				return results;
			}

			return files;
		}

		@Override
		public Class<?> getObjectType() {
			return Collection.class;
		}

		@Override
		public boolean isSingleton() {
			return false;
		}

		public void setDirectory(String dir) {
			this.directory = dir;
		}

		public void setFilters(Collection<FileFilter> filters) {
			this.filters = filters;
		}

		/**
		 * What we actually get from the processing of the nested tags
		 * is a collection of files within a collection so we flatten it and
		 * only keep the uniques
		 */
		public void setNestedFiles(Collection<Collection<File>> nestedFiles) {
			this.nestedFiles = new HashSet<File>(); // keep the list unique
			for (Collection<File> nested : nestedFiles) {
				this.nestedFiles.addAll(nested);
			}
		}

	}
}


You can see in the code above that I directly reference FileFilterDefinitionParser which is the part of the code that lets us filter any files that are in the list of files in the directory. Lets go into that code:

FileFilterDefinitionParser

package com.example.core.commons.tag;

import java.io.FileFilter;
import java.util.ArrayList;
import java.util.Collection;
import java.util.List;

import org.apache.commons.io.filefilter.NameFileFilter;
import org.springframework.beans.factory.FactoryBean;
import org.springframework.beans.factory.support.BeanDefinitionBuilder;
import org.springframework.beans.factory.support.ManagedList;
import org.springframework.beans.factory.xml.AbstractSingleBeanDefinitionParser;
import org.springframework.beans.factory.xml.ParserContext;
import org.w3c.dom.Element;
import org.w3c.dom.NodeList;

/**
 * Creates a list of FileFilters based on the configured value, ref, idRef, or bean.
 * <p/>
 *
 * Note: value defaults to a NamedFileFilter
 *
 * Example:
 *
 * Create a NamedFileFilter that will filter for the specific name
 *   <core-commons:fileFilter id="a">
 *       <value>someFile.txt</value>
 *   </core-commons:fileFilter>
 *
 * Create a filter that will filter for any file ending in .xml and that is writable
 *   <core-commons:fileFilter id="a">
 *       <bean class="org.apache.commons.io.filefilter.CanWriteFileFilter"/>
 *       <bean class="org.apache.commons.io.filefilter.RegexFileFilter">
 *           <constructor-arg value=".*.xml"/>
 *       </bean>
 *   </core-commons:fileFilter>
 *
 * @author seamans
 *
 */
public class FileFilterDefinitionParser
	extends AbstractSingleBeanDefinitionParser
{

	/**
	 * The bean that is created for this tag element
	 * 
	 * @param element The tag element
	 * @return A FileFilterFactoryBean
	 */
	@Override
	protected Class<?> getBeanClass(Element element) {
		return FileFilterFactoryBean.class;
	}

	/**
	 * Called when the fileFilter tag is to be parsed
	 * 
	 * @param element The tag element
	 * @param ctx The context in which the parsing is occuring
	 * @param builder The bean definitions build to use
	 */
	@Override
	protected void doParse(Element element, ParserContext ctx, BeanDefinitionBuilder builder) {
		
		// Set the scope
		builder.setScope(element.getAttribute("scope"));
		
		try {
			// All of the filters will eventually end up in this list
			// We use a 'ManagedList' and not a regular list because anything
			// placed in a ManagedList object will support all of Springs
			// functionalities and scopes for us, we dont' have to code anything
			// in terms of reference lookups, EL, etc
			ManagedList<Object> filters = new ManagedList<Object>();

			// For each child node of the fileFilter tag, parse it and place it
			// in the filtes list
			NodeList nl = element.getChildNodes();
			for (int i=0; i<nl.getLength(); i++) {
				DefinitionParserUtil.parseLimitedList(filters, nl.item(i), ctx, builder.getBeanDefinition(), element.getAttribute("scope"));
			}

			// Add the filtes to the list of properties (this is applied
			// to the factory beans setFilters below)
			builder.addPropertyValue("filters", filters);
		}
		catch (Exception e) {
			throw new RuntimeException(e);
		}
	}

	public static class FileFilterFactoryBean
		implements FactoryBean<Collection<FileFilter>>
	{

		private final List<FileFilter> filters = new ArrayList<FileFilter>();

		@Override
		public Collection<FileFilter> getObject() throws Exception {
			return filters;
		}

		@Override
		public Class<?> getObjectType() {
			return Collection.class;
		}

		@Override
		public boolean isSingleton() {
			return false;
		}

		/**
		 * Go through the list of filters and convert the String ones
		 * (the ones that were set with <value> and make them NameFileFilters
		 */
		public void setFilters(Collection<Object> filterList) {
			for (Object o : filterList) {
				if (o instanceof String) {
					filters.add(new NameFileFilter(o.toString()));
				}
				else if (o instanceof FileFilter) {
					filters.add((FileFilter)o);
				}
			}
		}

	}
}

As you saw in the XSD definition (previous post) for the fileFilter tag it supports child tags that are standard Spring tags (bean/ref/idref/value). We need to code support for this since unfortunately Spring does NOT provide a helper class for parsing its own tags. I was a bit disappointed in this and I have to admit that most of my time went into the coding of this util class and how I could support the Spring bean tag properly.

This piece of code may be of interest to others as it shows how to create a Spring bean programmatically and still get all its functionality

DefinitionParserUtil

package com.example.core.commons.tag;
package com.broadridge.adc.core.commons.tag;

import org.springframework.beans.factory.config.BeanDefinition;
import org.springframework.beans.factory.config.BeanDefinitionHolder;
import org.springframework.beans.factory.support.BeanDefinitionReaderUtils;
import org.springframework.beans.factory.support.DefaultListableBeanFactory;
import org.springframework.beans.factory.support.ManagedList;
import org.springframework.beans.factory.xml.BeanDefinitionParserDelegate;
import org.springframework.beans.factory.xml.ParserContext;
import org.springframework.expression.Expression;
import org.springframework.expression.ExpressionParser;
import org.springframework.expression.spel.standard.SpelExpressionParser;
import org.w3c.dom.Element;
import org.w3c.dom.Node;

public class DefinitionParserUtil {

	/**
	 * Parses the children of the passed in ParentNode for the following tags:
	 * <br/>
	 * value
	 * ref
	 * idref
	 * bean
	 * property
	 * *custom*
	 * <p/>
	 *
	 * The value tag works with Spring EL even in a Spring Batch scope="step"
	 *
	 * @param objects The list of resultings objects from the parsing (passed in for recursion purposes)
	 * @param parentNode The node who's children should be parsed
	 * @param ctx The ParserContext to use
	 * @param parentBean The BeanDefinition of the bean who is the parent of the parsed bean
	 * 		(i.e. the Bean that is the parentNode)
	 * @param scope The scope to execute in.  Checked if 'step' to provide Spring EL
	 * 		support in a Spring Batch env
	 * @throws Exception
	 */
	public static void parseLimitedList(ManagedList<Object> objects, Node node,
		ParserContext ctx, BeanDefinition parentBean, String scope)
		throws Exception
	{
		parseLimitedList(objects, node, ctx, parentBean, scope, true);
	}

	/**
	 * Parses the children of the passed in ParentNode for the following tags:
	 * <br/>
	 * value
	 * ref
	 * idref
	 * bean
	 * property
	 * *custom*
	 * <p/>
	 *
	 * The value tag works with Spring EL even in a Spring Batch scope="step"
	 *
	 * @param objects The list of resultings objects from the parsing (passed in for recursion purposes)
	 * @param parentNode The node who's children should be parsed
	 * @param ctx The ParserContext to use
	 * @param parentBean The BeanDefinition of the bean who is the parent of the parsed bean
	 * 		(i.e. the Bean that is the parentNode)
	 * @param scope The scope to execute in.  Checked if 'step' to provide Spring EL
	 * 		support in a Spring Batch env
	 * @param supportCustomTags Should we support custom tags within our tags?
	 * @throws Exception
	 */
	public static void parseLimitedList(ManagedList<Object> objects, Node node,
		ParserContext ctx, BeanDefinition parentBean, String scope, boolean supportCustomTags)
		throws Exception
	{
		// Only worry about element nodes
		if (node.getNodeType() == Node.ELEMENT_NODE) {
			Element elem = (Element)node;
			String tagName = node.getLocalName();

			if (tagName.equals("value")) {
				String val = node.getTextContent();
				// to get around an issue with Spring Batch not parsing Spring EL
				// we will do it for them
				if (scope.equals("step")
					&& (val.startsWith("#{") && val.endsWith("}"))
					&& (!val.startsWith("#{jobParameters"))
					)
				{
					// Set up a new EL parser
					ExpressionParser parser = new SpelExpressionParser();
					// Parse the value
					Expression exp = parser.parseExpression(val.substring(2, val.length()-1));
					// Place the results in the list of created objects
					objects.add(exp.getValue());
				}
				else {
					// Otherwise, just treat it as a normal value tag
					objects.add(val);
				}
			}
			// Either of these is a just a lookup of an existing bean 
			else if (tagName.equals("ref") || tagName.equals("idref")) {
				objects.add(ctx.getRegistry().getBeanDefinition(node.getTextContent()));
			}
			// We need to create the bean
			else if (tagName.equals("bean")) {
				// There is no quick little util I could find to create a bean
				// on the fly programmatically in Spring and still support all
				// Spring functionality so basically I mimic what Spring actually
				// does but on a smaller scale.  Everything Spring allows is
				// still supported

				// Create a factory to make the bean
				DefaultListableBeanFactory beanFactory = new DefaultListableBeanFactory();
				// Set up a parser for the bean
				BeanDefinitionParserDelegate pd = new BeanDefinitionParserDelegate(ctx.getReaderContext());
				// Parse the bean get its information, now in a DefintionHolder
				BeanDefinitionHolder bh = pd.parseBeanDefinitionElement(elem, parentBean);
				// Register the bean will all the other beans Spring is aware of
				BeanDefinitionReaderUtils.registerBeanDefinition(bh, beanFactory);
				// Get the bean from the factory.  This will allows Spring
				// to do all its work (EL processing, scope, etc) and give us 
				// the actual bean itself
				Object bean = beanFactory.getBean(bh.getBeanName());
				objects.add(bean);
			}
			/*
			 * This is handled a bit differently in that it actually sets the property
			 * on the parent bean for us based on the property
			 */
			else if (tagName.equals("property")) {
				BeanDefinitionParserDelegate pd = new BeanDefinitionParserDelegate(ctx.getReaderContext());
				// This method actually set eh property on the parentBean for us so
				// we don't have to add anything to the objects object
				pd.parsePropertyElement(elem, parentBean);
			}
			else if (supportCustomTags) {
				// handle custom tag
				BeanDefinitionParserDelegate pd = new BeanDefinitionParserDelegate(ctx.getReaderContext());
				BeanDefinition bd = pd.parseCustomElement(elem, parentBean);
				objects.add(bd);
			}
		}
	}
}

And that’s it! Ok, so you may be saying, “Whoa, that was a lot of code”, well, I did say it was a more advanced example.

If you have any questions please post a comment. I know it can be a bit confusing so I tried to document in-line in the code as much as possible to make it easy to follow.

Really though, this is a very powerful, under-utilized feature of Spring. Knowing how to do this can really make your XML smaller while harnessing the power of converting things on the fly and keeping your beans as generic as possible.

Posted in Java, Programming | Tagged , | 4 Comments

Spring Custom Tags (Extensible XML) – Part 1

***** WARNING – This is my Kill Bill post (was one post, was too long, is now two posts) *****

I’m surprised by how many people don’t use a nice little ability in Spring, the custom tag support (or as they call it, Extensible XML Authoring). Too many times you end up with drawn out bean definitions that are all flavors on a theme or you end up writing beans that do nothing but convert one object type to another to get it into some other bean object.

A great example of this is the example that Spring uses in its documentation on creating a custom tag, creating a SimpleDateFormat external to an existing bean. I won’t go into it too deeply (feel free to read it at http://static.springsource.org/spring/docs/current/spring-framework-reference/html/extensible-xml.html) but Spring allows you to write a tag that will create a SimpleDateFormat in one tag. Ok, maybe this does require a bit of explanation. See, in Spring you would normally do something like this:

<bean id="myFormatter" class="java.text.SimpleDateFormat">
    <constructor-arg value="yyyyMMdd"/>
</bean>

Now, that is all well and good, but with custom tags you could write some code and just do:

<mytag:dateFormat format="yyyyMMdd"/>

Short, sweet, and to the point.

Level up!

The Spring docs also give examples of slight more complex examples, but not really anything useful. So let me show you something I did for a project I’m on that allows us to easily send List into beans in a very cool (well, ok, I think it cool) and easy way while also providing file filtering. The purpose of this example is to show you what can really be done with custom tags and how easy it is once you understand all the moving parts.

There are a few parts to writing a custom tag:

  • The Spring Configuration for the Tags
  • The Namespace Handler
  • The XSD file
  • The DefinitionParser

I’m going to present them in the order above as I think it the most logical and easy to follow.

Note: For the sake of this example I’m going to put everything in the XML Namespace www.example.com/core-commons. If you don’t know what Namespacing is in XML its (and this is simplfying it as much as I can) a way to say that certain tags are in a certain URL domain. This makes sure that if you have to create and XML document that uses one schema that contains a bean and another schema that also contains a bean you can do so and still keep the bean tags separate. Like I said, I tried to simplify it and I think I may have failed…

The Spring Configuration for the Tags

To get custom tags to work you first have to let Spring know that they exist. This is actually quite simple. First, create a file in the root of your META-INF named spring.schemas and place the Namespace of your tag and the .xsd’s location in it:

http\://www.example.com/schema/core-commons-1.0.xsd=com/example/core/commons/tag/core-commons-1.0.xsd

Few things to note:

  • The http\: is required to get around the way things are parsed by Spring. You NEED that \ before the :
  • There is nothing wrong with placing the xsd inside a package structure. I prefer it as it ensured you don’t collide ont he classpath with another file of the same name from a different package
  • Always put a version number on your xsd. That way you can control versions at a later date

So that told Spring where it can find the XSD file to enfore the XML validation rules. Next we need to tell Spring where it can find the code to handle the different tags you may have created. This is placed in a file in the root of your META-INF named spring.handlers and looks like this:

http\://www.example.com/schema/core-commons-1.0.xsd=com.example.core.commons.tag.CoreNamespaceHandler

As you can see the Namespace definition is the same as above but now we has specified a class that will handle the Namespace for us (should Namespace be capitalized like that? I'm not sure, lets stick with it for now shall we?)

The Namespace Handler

The NamespaceHandler (defined as com.example.core.commons.tag.CoreNamespaceHandler above) tells Spring what code to execute for whatever tag is used. The easiest way to do this is to extend org.springframework.beans.factory.xml.NamespaceHandlerSupport. For this example where we will support sending List into beans my code looks like:

package com.example.core.commons.tag;

import org.springframework.beans.factory.xml.NamespaceHandlerSupport;

public class CoreNamespaceHandler
    extends NamespaceHandlerSupport
{

    @Override
    public void init() {
        this.registerBeanDefinitionParser("fileList", new FileListDefinitionParser());
        this.registerBeanDefinitionParser("fileFilter", new FileFilterDefinitionParser());
    }
}

What the code is doing is saying "for the tag 'fileList' parse it with the object FileListDefinitionParser". Not much else to it. See, I told you custom tags are easy! Well, so far anyway.

The XSD file

So, you have it so Spring knows where things are and what it should fire when it hits the custom tags, but what do the tags actually look like? What do they support in terms of child elements and attributes?

For that you need to define the .xsd for the tag. We already defined the name and location of the .xsd tag above in the spring.schemas file. Now lets make it. First, create the .xsd in the correct location (com/example/core/commons/tag) and name it core-commons-1.0.xsd. In this file place your xsd. For this example it's :

<?xml version="1.0" encoding="UTF-8"?>
<xsd:schema xmlns="http://www.example.com/schema/core-commons-1.0" 
	targetNamespace="http://www.example.com/schema/core-commons-1.0" 
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
	xmlns:xsd="http://www.w3.org/2001/XMLSchema" 
	xmlns:beans="http://www.springframework.org/schema/beans" 
	elementFormDefault="qualified" 
	attributeFormDefault="unqualified"
	version="1.0">
	
	<xsd:import namespace="http://www.springframework.org/schema/beans" schemaLocation="http://www.springframework.org/schema/beans/spring-beans-3.0.xsd"/>
	
    <xsd:element name="fileList">
        <xsd:complexType>
            <xsd:complexContent>
                <xsd:extension base="beans:identifiedType">
                    <xsd:sequence>
                        <xsd:element ref="fileFilter" minOccurs="0" maxOccurs="1"/>
                        <xsd:element ref="fileList" minOccurs="0" maxOccurs="unbounded"/>
                    </xsd:sequence>
                    <xsd:attribute name="directory" type="xsd:string"/>
                    <xsd:attribute name="scope" type="xsd:string"/>
                </xsd:extension>
            </xsd:complexContent>
        </xsd:complexType>
    </xsd:element>
    
    <xsd:element name="fileFilter">
        <xsd:complexType>
            <xsd:complexContent>
                <xsd:extension base="beans:identifiedType">
                    <xsd:group ref="limitedType"/>
                    <xsd:attribute name="scope" type="xsd:string"/>
                </xsd:extension>
            </xsd:complexContent>
        </xsd:complexType>
    </xsd:element>

    <xsd:group name="limitedType">
        <xsd:sequence>
            <xsd:choice minOccurs="1" maxOccurs="unbounded">
                <xsd:element ref="beans:bean"/>
                <xsd:element ref="beans:ref"/>
                <xsd:element ref="beans:idref"/>
                <xsd:element ref="beans:value"/>
                <xsd:any minOccurs="0"/>
            </xsd:choice>
        </xsd:sequence>
    </xsd:group>
</xsd:schema>

Ok, now, if you don't know XSD you may be pretty lost at this point and I would be to so let me try to put what is going on in simple terms without taking the time to teach XSD (there are many tutorials on that already). I'm going to do this by line and I'll skip the lines that really are not that important but you should keep in

<xsd:schema xmlns="http://www.example.com/schema/core-commons-1.0" 

The default Namespace of this document

targetNamespace="http://www.example.com/schema/core-commons-1.0" 

Define the Namespace this file represents. It should match what you put in spring.schemas (this first part)

xmlns:beans="http://www.springframework.org/schema/beans" 

On line 11 we pull in the Spring XSD. So we have access to its tags we place it in the beans Namespace

<xsd:import namespace="http://www.springframework.org/schema/beans" schemaLocation="http://www.springframework.org/schema/beans/spring-beans-3.0.xsd"/>

Pull in the Spring XSD file

<xsd:element name="fileList">

Define our first tag, fileList

<xsd:extension base="beans:identifiedType">

Extend Springs identifiedType. This, in Spring, is what decides if a tag can have an id attribute on it. Remember that nested Spring beans can't have the id tag on them. This helps with that

<xsd:element ref="fileFilter" minOccurs="0" maxOccurs="1"/>

Allow a child tag named fileFilter that can occur 0..1 times

<xsd:element ref="fileList" minOccurs="0" maxOccurs="unbounded"/>

Allow a child tag named fileList (hey! That's what we are doing now. We're going recursive! Woot!) that can occur 0..N times

<xsd:attribute name="directory" type="xsd:string"/>

directory. This will allow us to define the directory we want to read files from

<xsd:attribute name="scope" type="xsd:string"/>

Define an attribute named scope. This will allow us to set the scope of the bean (Later in the code you will see that I also support Spring Batch’s step scope

<xsd:element name="fileFilter">

Define a tag named fileFilter. Since we have defined it this way it can be used on its own or, since like 18 defined it as a child of fileList, it can be used there as well

<xsd:group ref="limitedType"/>

Bring in a group of already defined (see line 39) tags that we will support as children to fileFilter

<xsd:group name="limitedType">

Define a group of tags for use in other tags

<xsd:element ref="beans:bean"/>
<xsd:element ref="beans:ref"/>
<xsd:element ref="beans:idref"/>
<xsd:element ref="beans:value"/>

Add support for a set of Spring tags. This is the main reason we imported the Spring XSD on line 11

<xsd:any minOccurs="0"/>

any allows us to support any other tag that may be defined (additional custom tags perhaps?)

I hope that wasn’t too hard to follow. If you know XSD it shouldn’t have been. If you don’t know XSD and are still very confused please ask questions and I’ll do my best to answer.

This concludes Part 1. In Part two I’ll go into the code that is actually involved 🙂

Posted in Java, Programming | Tagged , | 3 Comments

Dynamic Configuration of Hibernate Envers

The project I’m working on wanted to have audit tables for many of the tables in the system. Instead of writing our own plugins and creating the tables we decided to use Hibernate Envers.

Hibernate Envers is a really nice addition to Hibernate that allows you to flag domain objects with @Audited and it will do all the work for you.

The Problem

The problem with Envers is in its configuration. Being that it is a JPA solution, it sticks to the JPA standard very tightly. Well, what’s wrong with that? Things should stick to the standard. I agree with that. The thing is (in my best Jeremy Clarkson voice) is that this means all the configuration is in the persistence.xml file.

Now, this is fine and dandy in a set environment where you know what you will be deploying to and how things will be configured. But what if you don’t? Well, the standard convention is to create an external properties file and have Spring read the properties file when it starts up and replace all of your ${} with the variables in the external properties files.

This allows things like DB connection information and such to be external to the app. The issue is that this variable replacement doesn’t exist in persistence.xml (I hear that it may be in the next spec) and that Envers does something a little odd. What it does that is odd is that it uses the connection information that it picks up from your Hibernate configuration (which you could set with ${} when you configured your dataSource in your Spring configuration) but it doesn’t use the same schema. Envers defaults to the schema of the user who logged into the db and well, that might not be the same schema you have Hibernate using or even what you want.

To address this, Envers allows you to set the schema to use with the org.hibernate.envers.default_schema property. Great! But wait, it’s set in the persistence.xml. You know, the thing you can’t use the external properties file against. This means that you have to hard code the schema you want to use. So much for external configuration you say. Well, not really

The Solution

Luckily Spring supports registering a post-processor of the persistence.xml. The LocalContainerEntityManagerFactoryBean allows you to set a persistenceUnitPostProcessors property that allows you to manipulate the persistence.xml before it is really used.

So, what we can do is create an object that takes properties such as “schema” and sets the org.hibernate.envers.default_schema property for us. Since we are doing this in Spring we can leverage its ${} replacement and set the schema from the external properties file. Woot!

First things first, lets create our object:

import org.springframework.orm.jpa.persistenceunit.MutablePersistenceUnitInfo;
import org.springframework.orm.jpa.persistenceunit.PersistenceUnitPostProcessor;

/**
 * Allows for the modification of Hibernate Envers configuration via external
 * properties files instead of being hard-coded into the persistence.xml
 * <p/>
 *
 * Should be added onto
org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean
 *
 * <p/>
 * Example:
 *     &lt;bean id="x"
class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean"&gt;
 *        &lt;property name="dataSource" ref="dataSource"/&gt;
 *        &lt;property name="jpaVendorAdapter"&gt;
 *            &lt;bean
class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter"&gt;
 *                &lt;property name="showSql" value="false"/&gt;
 *                &lt;property name="databasePlatform"
value="org.hibernate.dialect.SQLServerDialect"/&gt;
 *                &lt;property name="generateDdl" value="true"/&gt;
 *            &lt;/bean&gt;
 *        &lt;/property&gt;
 *        &lt;property name="persistenceUnitName" value="model"/&gt;
 *        &lt;property name="persistenceUnitPostProcessors"&gt;
 *            &lt;list&gt;
 *                &lt;bean
class="com.broadridge.adc.fundmap.batch.EnversPersistenceUnitPostProcessor"&gt;
 *                    &lt;property name="schema" value="SCHEMA_NAME"/&gt;
 *                &lt;/bean&gt;
 *            &lt;/list&gt;
 *        &lt;/property&gt;
 *     &lt;/bean&gt;
 * @author seamans
 *
 */
public class EnversPersistenceUnitPostProcessor
       implements PersistenceUnitPostProcessor {

       private static final String ENVERS_SCHEMA =
"org.hibernate.envers.default_schema";
       private static final String ENVERS_CATALOG =
"org.hibernate.envers.default_catalog";
       private static final String ENVERS_TRACK_CHANGE =
"org.hibernate.envers.track_entities_changed_in_revision";
       private static final String ENVERS_TABLE_SUFFIX =
"org.hibernate.envers.audit_table_suffix";
       private static final String ENVERS_TRACK_COLLECTION_CHANGE =
"org.hibernate.envers.revision_on_collection_change";
       private static final String ENVERS_NO_OPT_LOCKING =
"org.hibernate.envers.do_not_audit_optimistic_locking_field";
       private static final String ENVERS_STORE_ON_DELETE =
"org.hibernate.envers.store_data_at_delete";
       private static final String ENVERS_REVISION_LISTENER =
"org.hibernate.envers.revision_listener";

       private String schema;
       private String catalog;
       private String trackChange;
       private String suffix;
       private String trackCollectionChange;
       private String noOptLocking;
       private String storeOnDelete;
       private String revisionListener;

       @Override
       public void postProcessPersistenceUnitInfo(MutablePersistenceUnitInfo pui) {
               if (schema != null) {
                       pui.addProperty(ENVERS_SCHEMA, schema);
               }
               if (catalog != null) {
                       pui.addProperty(ENVERS_CATALOG, catalog);
               }
               if (trackChange != null) {
                       pui.addProperty(ENVERS_TRACK_CHANGE, trackChange);
               }
               if (suffix != null) {
                       pui.addProperty(ENVERS_TABLE_SUFFIX, suffix);
               }
               if (trackCollectionChange != null) {
                       pui.addProperty(ENVERS_TRACK_COLLECTION_CHANGE, trackCollectionChange);
               }
               if (noOptLocking != null) {
                       pui.addProperty(ENVERS_NO_OPT_LOCKING, noOptLocking);
               }
               if (storeOnDelete != null) {
                       pui.addProperty(ENVERS_STORE_ON_DELETE, storeOnDelete);
               }
               if (revisionListener != null) {
                       pui.addProperty(ENVERS_REVISION_LISTENER, revisionListener);
               }
       }

       public void setSchema(String schema) {
               this.schema = schema;
       }

       public void setCatalog(String catalog) {
               this.catalog = catalog;
       }

       public void setTrackChange(String trackChange) {
               this.trackChange = trackChange;
       }

       public void setSuffix(String suffix) {
               this.suffix = suffix;
       }

       public void setTrackCollectionChange(String trackCollectionChange) {
               this.trackCollectionChange = trackCollectionChange;
       }

       public void setNoOptLocking(String noOptLocking) {
               this.noOptLocking = noOptLocking;
       }

       public void setStoreOnDelete(String storeOnDelete) {
               this.storeOnDelete = storeOnDelete;
       }

       public void setRevisionListener(String revisionListener) {
               this.revisionListener = revisionListener;
       }

}

Basically, all I’ve done is write a PersistenceUnitPostProcessor that allows us to set properties based on standard bean conventions.

Now we can leverage Spring and set the properties from an external properties file:

   <bean id="entityManager"
       class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
       <property name="dataSource" ref="dataSource"/>
       <property name="jpaVendorAdapter">
           <bean
class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter">
               <property name="showSql" value="false"/>
               <property name="databasePlatform"
value="org.hibernate.dialect.SQLServerDialect"/>
               <property name="generateDdl" value="true"/>
           </bean>
       </property>
       <property name="persistenceUnitName" value="model"/>
       <property name="persistenceUnitPostProcessors">
           <list>
               <bean
class="EnversPersistenceUnitPostProcessor">
                   <property name="schema" value="${dataSource.audit.schema}"/>
               </bean>
           </list>
       </property>
   </bean>

See the ${dataSource.audit.schema}? Just put dataSource.audit.schema in your external properties file and you will be able to control the schema name from outside the application!

You will notice that my object also supports setting other properties. I wrote code for all of the other properties that I could find in case I needed them at a later time.

That’s it! Envers can now be configures from an external property file. Enjoy!

Posted in Java, Programming | Tagged , , , , | Leave a comment

Groovy method overriding

One of the pieces of the project I am currently working on is using Grails and GORM for rapid deployment. In one of the GORM objects I wanted to override one of the methods that Groovy creates for a property so I can derive some information from the property at the time it was set.

It sounds simple enough but, like most things once you dive into it, it wasn’t. Even after a good bit of Googling I could only find partial answers or comments in posts that led you part way to the solution.

So, let me present to you the situation in more detail and the solution so you don’t have to do all the digging and Googling that I had to.

Lets begin with a sample GORM object

class Person {
    String firstName
    String lastName
}

Simple enough. Now, what if we wanted to add a property like isLastNameSmith. How would you do it?

Well one way would be:

class Person {
    String firstName
    String lastName
    boolean lastNameSmith = false
}

That would work but you are now forcing the developer to check the last name and relying on them to set the boolean. In a large scale project with many developers, this would not be a chance I would want to take. It would be too easy to forget to set the boolean.

Ok, so you decide “Well, I’ll set it for them when the lastName is set”. Good idea! Not only to you take the burden off of the developer, if the value changes in the DB the flag will be correct as soon as the object is loaded.

So, what’s involved in this little slick solution? Well, first, we have to make lastNameSmith private so that the developer can’t change it. Well, if we make it private we have to provide a method to check it. Lets stop here and write that code:

class Person {
    String firstName
    String lastName
    private boolean lastNameSmith = false

    def isLastNameSmith() {
        return lastNameSmith
    }
}

Great! You now have locked down the access. All you need to do is check if the last name is ‘Smith’ when the last name is set. So, you think to yourself “Easy enough, I’ll just do:”

    def setLastName(s) {
        lastName = s
        if (lastName == 'Smith') lastNameSmith = true  // Lets be case sensitve to keep the example easy
    }

And…. you’d be half right and just lucky at the same time. Wait, what? Sorry, but it wouldn’t catch all the ways to set last name. Well, why not?

First, lets remember that you can set a property in Groovy in two ways:

  • method call – object.setLastName(“x”)
  • direct setting – object.lastName = “x”

In Groovy you need to handle both cases. Lets address the first, the method call.

If you code:

    def setLastName(s) {}

Groovy generates:

    public Object setLastName(Object s) {}

Now, you got lucky. Your code, when calling object.setLastName("x") will match the generated Groovy method. But it might not always do that. Say you want to override a boolean setter like so:

    def setIsSomething(boolValue) 

Groovy generates:

    pubic Object setIsSomething(Object boolValue)

This would not be what you wanted because you are actually looking for:

    public void setIsSomething(boolean boolValue)

So, what you need to do is explicitly override the exact java syntax method call to ensure that your code gets called. If you don’t do that you risk the possibility in certain instances of not having the methods match and your override method not being called.

Next is the other way Groovy allows you to easily set properties, via direct reference like object.lastNameSmith. Now yes, you could do this in standard Java as well though it is advised against for maintenance reasons unless you are really concerned about performance (it is one of the performance optimizations suggested for Android development by Google).

In this case you may thing to yourself, well, now I’m screwed. And you’d be wrong again! (Hey, making mistakes is sometimes the best way to learn ;)). Groovy doesn’t directly set the property when invoked like object.lastNameSmith, it calls a method it creates named setProperty(String name, Object value). Since we know Groovy is going to do this, we can override that method and catch the property call to lastName!

So, the second part of the solution would look like:

   public void setProperty(String name, Object value) {
       def metaProperty = this.metaClass.getMetaProperty(name)
       if(metaProperty) metaProperty.setProperty(this, value)
       if (name == "lastName" && value? == "Smith") this.isLastNameSmith = true;
   }

The meta stuff is just what Groovy does to normally set a property. We still need to do this to support all the other properties that will be set as well as handle the case we are looking for.

So, our final object would be:

public Person {
    String firstName
    String lastName
    private boolean lastNameSmith = false

    def isLastNameSmith {
        return lastNameSmith
    }

    public void setLastName(String s) {
        this.lastName = s;
        if (s == "Smith") lastNameSmith = true;
    }

    pubic void setProperty(String name, Object value) {
        def metaProperty = this.metaClass.getMetaProperty(name)
        if(metaProperty) metaProperty.setProperty(this, value)
        if (name == "lastName" && value? == "Smith") this.lastNameSmith = true;
   }

This solution covers both ways of a property being set and ensures that the developer doesn’t have to worry about checking the last name to set a boolean as well as making sure that any changes in the database will be reflected in the object.

Posted in Groovy | Tagged , | Leave a comment