Sunday, June 3, 2012

Daogen a Generator for JPA Dao Implementations

Recently I created a small library to generate Java JPA Dao's from Dao interface type annotated with query string. There are no runtime dependencies other the javax.persistence. Compile-time dependencies are freemarker and jalopy.
One of the drawbacks of using Named Queries in JPA is that you have to attach the query definitions either as annotation to an Entity class or put them manually in separate orm mapping files (XML). Daogen lets you attach query string as annotation to a method in your Dao interface type like this:
@DaoTypepublic interface CustomerDao {        
   @JpaNamedQuery(query = "select c from Customer c where name = :name")    
   List<Customer> findByName(String name);    

   @JpaNamedQuery(query = "select c from Customer c where name = :name
        and c.creationDate < :date", throwException = false)    
   Customer findByUniqueNameAndBeforeDate(String name, @JpaTemporal(TemporalType.DATE)Date date);    
   
   @JpaNamedQuery(query = "select c from Customer c where name = :name", throwException = true)    
   Customer findByUniqueNameMandatory(String name);    
   
   @JpaNamedQuery(query = "select o from Order o where o.customer.id = :id")    
   List<Order> findOrdersByCustomerId(Long id);    
}
Daogen will create Dao implementations for all Dao interface types, in this case CustomerDaoImpl. Method parameters are matched against query parameter names. If you don't want named queries you can use dynamic queries by using @JpaQuery instead of @JpaNamedQuery. Furthermore it generates one queries-orm.xml file for all named queries.
@Generated(value="Generated by nl.koelec.daogen.gen.DaoProcessor", date = "Mon May 28 21:43:32 CEST 2012")
public class CustomerDaoImpl extends AbstractDao implements CustomerDao {    
   private final EntityManager entityManager;    

   public CustomerDaoImpl(EntityManager entityManager) {        
       this.entityManager = entityManager;    
   }    

   @Override        
   protected EntityManager getEntitymanager() {
       return entityManager;        
   }
    
   @Override        
   public List findByName(String name) {
      Query query = getEntitymanager().createNamedQuery("CustomerDao.findByName"); 
      setParam("name", name, null, query);            
      return query.getResultList();        
   }
   ...
}

queries-orm.xml:

<entity-mappings xmlns="http://java.sun.com/xml/ns/persistence/orm"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation=" 
    http://java.sun.com/xml/ns/persistence/orm 
    http://java.sun.com/xml/ns/persistence/orm_1_0.xsd"version="1.0">
<!-- @Generated(value="Generated by nl.koelec.daogen.gen.DaoProcessor",
date = "Mon May 28 21:43:32 CEST 2012") -->
<named-query name="CustomerDao.findByName"> <query><![CDATA[ select c from Customer c where name = :name ]]></query></named-query>
<named-query name="CustomerDao.findByUniqueNameAndBeforeDate"> <query><![CDATA[ select c from Customer c where name = :name and c.creationDate < :date ]]></query></named-query>
...
</entity-mappings>

The queries-orm file is referenced from persistence.xml like so:
<persistence-unit name="testPU" transaction-type="RESOURCE_LOCAL">
   <provider>org.hibernate.ejb.HibernatePersistence</provider>
   <mapping-file>queries-orm.xml</mapping-file>
   ...

GITHUB

The code is in github and can be build with gradle
There are three gradle projects: daogen-core, dao-gen-processor and daogen-sample, each containing just a couple of java class files and some freemarker template files (part of daogen-processor).At runtime you will only need daogen-core. The daogen-processor jar contains the AnnotationProcessor used to generate the files at compile time.
You can use the prebuild jar files from the dist folder right away. Or you can build the projects yourself and go from there. The gradlew.bat file in the base folder lets you build daogen from source, whithout installing anything in advance with the following invocation: gradlew.bat build
After this you can take a look at the generated code and the test code in daogen-sample.

Using daogen in your own project.

Maven

put this plugin config in your pom file.
<build>   
   <plugins>      
      <plugin>         
         <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
              <version>2.3.2</version>              
              <configuration>                 
                 <source>1.6</source>                 
                 <target>1.6</target>                 
                 <generatedSourcesDirectory>generated-sources/src/main/java</generatedSourcesDirectory><annotationProcessors>
              <annotationProcessor>nl.koelec.daogen.gen.DaoProcessor</annotationProcessor>  
              </annotationProcessors>              
        </configuration>      
      </plugin>   
   </plugins>
</build>
Of course you also need the daogen-core and daogen-processor.jar as dependencies in your pom file.

Gradle

Look at gradle.build file in daogen-sample project.

Configuration

If you want you can override the used freemarker templates to change or add behaviour.Without supplying options to the DaoProcessor it will use the default templates packaged with the daogen-processor jar file.The following DaoProcessor options are supported:
  • templatedir folder where to look for override templates
  • ormtemplate queries orm template file name
  • daotemplate dao implementation template file name
  • daofactorytemplate dao factory implementation template file name
  • idaofactorytemplate dao factory interface template file name

Note

Don't forget to mark your Dao interface type with @DaoType at the class level.

Filter expressions in Rest urls leveraging QueryDsl Part 2

In my previous post I showed how to use QueryDsl-like BooleanExpressions as value of a Url query-parameter, as in this example:
http://www.myprettyproducts.com/product?filter="type.eq(3).and(prize.lt(500))"
The proposed solution leveraged both MVEL and QueryDsl to convert the filter expression at runtime to an executable QueryDsl expression.
As indicated in my post this approach introduced a security vulnerability as we open up our application to code injection via the filter expression. The filter expression is just a string which is interpreted by MVEL as Java code.
The way I defeated this is to restrict the expression string before it is given to MVEL. It is restricted to only accept terms and constructions that lead to a valid QueryDsl Predicate and nothing more. For this purpose I wrote a custom dynamic Parser using Parboild. Parboild is a socalled PEG (Parsing expression grammar) parser.
This is the grammar I came up with to parse the expressions:
FilterExpression : Expression
Expression       : MethodInvocation ['.'MethodInvocation]+
MethodInvocation : Method ArgumentList
Method           : Path
Path             : PathElement ['.'PathElement]+
PathElement      : 
ArgumentList     : '(' Argument? ['.'Argument]+ ')'
Argument         : Literal / Expression / Path / Array
Literal          : FloatLiteral / IntegerLiteral / CharLiteral
                    / StringLiteral / StringLiteral2 / 'true' / 'false' / 'null'
Array            : '[' (Literal / [','Literal]+) / (Expression / [','Expression]+)  ']'
The rule definition of the literal values is copied from an example for parsing of Java source code from the Parboiled website.
public class FilterExpressionParser extends BaseParser<Object> {
    private String[] allowedPathElements;

    public FilterExpressionParser(Collection<String> allowedPathElements) {
        this.allowedPathElements = (String[])allowedPathElements.toArray(new String[0]);
        Arrays.sort(this.allowedPathElements, new Comparator<String>() {
            @Override
            public int compare(java.lang.String o1, java.lang.String o2) {
                return o2.compareTo(o1);
            }
        });
    }

    Rule FilterExpression() {
        return Sequence(Expression(), EOI);
    }

    Rule Expression() {
        return Sequence(MethodInvocation(), ZeroOrMore(".", MethodInvocation()));
    }

    Rule MethodInvocation() {
        return Sequence(Method(), ArgumentList());
    }

    Rule ArgumentList() {
        return Sequence("(", Optional(Argument(), ZeroOrMore(",", Argument())), ")");
    }

    Rule Argument() {
        return FirstOf(Literal(), Expression(), Path(), Array());
    }

    Rule Array() {
        return Sequence("[",
                FirstOf(Sequence(Literal(), ZeroOrMore(",", Literal())), 
                Sequence(Expression(), ZeroOrMore(",", Expression()))),
                "]");
    }

    Rule Method() {
        return Path();
    }

    Rule Path() {
        return Sequence(FirstOf(allowedPathElements), ZeroOrMore(".", FirstOf(allowedPathElements)));
    }

    @MemoMismatches
    Rule LetterOrDigit() {
        // switch to this "reduced" character space version for a ~10% parser performance speedup
        return FirstOf(CharRange('a', 'z'), CharRange('A', 'Z'), CharRange('0', '9'), '_', '$');
        // return FirstOf(Sequence('\\', UnicodeEscape()), new JavaLetterOrDigitMatcher());
    }

    Rule Literal() {
        return FirstOf(Sequence(Optional("-"), FloatLiteral()), Sequence(Optional("-"),
                IntegerLiteral()), CharLiteral(),
                StringLiteral(), StringLiteral2(), Sequence("true", TestNot(LetterOrDigit())),
                Sequence("false", TestNot(LetterOrDigit())), Sequence("null", TestNot(LetterOrDigit()))

        );
    }

    @SuppressSubnodes
    Rule IntegerLiteral() {
        return Sequence(DecimalNumeral(), Optional(AnyOf("lL")));
    }

    @SuppressSubnodes
    Rule DecimalNumeral() {
        return FirstOf('0', Sequence(CharRange('1', '9'), ZeroOrMore(Digit())));
    }

    Rule HexDigit() {
        return FirstOf(CharRange('a', 'f'), CharRange('A', 'F'), CharRange('0', '9'));
    }

    Rule FloatLiteral() {
        return DecimalFloat();
    }

    @SuppressSubnodes
    Rule DecimalFloat() {
        return FirstOf(Sequence(OneOrMore(Digit()), '.', ZeroOrMore(Digit()), 
                Optional(Exponent()), Optional(AnyOf("fFdD"))),
                Sequence('.', OneOrMore(Digit()), Optional(Exponent()), Optional(AnyOf("fFdD"))),
                Sequence(OneOrMore(Digit()), Exponent(), Optional(AnyOf("fFdD"))),
                Sequence(OneOrMore(Digit()), Optional(Exponent()), AnyOf("fFdD")));
    }

    Rule Exponent() {
        return Sequence(AnyOf("eE"), Optional(AnyOf("+-")), OneOrMore(Digit()));
    }

    Rule Digit() {
        return CharRange('0', '9');
    }

    Rule CharLiteral() {
        return Sequence('\'', FirstOf(Escape(), Sequence(TestNot(AnyOf("'\\")), ANY)).suppressSubnodes(), '\'');
    }

    Rule StringLiteral() {
        return Sequence('"', ZeroOrMore(FirstOf(Escape(), 
               Sequence(TestNot(AnyOf("\r\n\"\\")), ANY))).suppressSubnodes(), '"');
    }

    Rule StringLiteral2() {
        return Sequence('\'', ZeroOrMore(FirstOf(Escape(), 
               Sequence(TestNot(AnyOf("\r\n'\\")), ANY))).suppressSubnodes(), '\'');
    }

    Rule Escape() {
        return Sequence('\\', FirstOf(AnyOf("btnfr\"\'\\"), OctalEscape(), UnicodeEscape()));
    }

    Rule OctalEscape() {
        return FirstOf(Sequence(CharRange('0', '3'), CharRange('0', '7'), CharRange('0', '7')),
                Sequence(CharRange('0', '7'), CharRange('0', '7')), CharRange('0', '7'));
    }

    Rule UnicodeEscape() {
        return Sequence(OneOrMore('u'), HexDigit(), HexDigit(), HexDigit(), HexDigit());
    }
}
You use it like this:
    private final static List queryDslMethods = Arrays.asList("and", "or", "not", "eq", "ne", 
"in", "notIn", "after", "before", "between", "notBetween", "lt", "loe", "gt", "goe",
"equalsIgnoreCase", "like", "matches", "startsWith", "startsWithIgnoreCase"); public Predicate toPredicate() { // create set with all terms that are allowed to appear as 'methods' in a filter expression Set allowedMethods = new HashSet(); set.addAll(queryDslMethods); set.addAll(Arrays.asList("type", "prize", "creationDate")); set.add("_date"); set.add("valueOf"); // needed for ValueFactory calls, i.e. _date.valueOf(...) // create the Parboiled parser with our FilterExpressionParser class FilterExpressionParser parser = Parboiled.createParser(FilterExpressionParser.class, allowedMethods); ReportingParseRunner<Object> parseRunner = new ReportingParseRunner<Object>(parser.FilterExpression()); // run the parser and examine result ParsingResult<Object> parsingResult = parseRunner.run(expression); if (!parsingResult.matched) { throw new IllegalArgumentException("filter expression is invalid: " + expression); } return evalExpression(expression); } private Predicate evalExpression(String expression) // create a map with all Objects that need to be available in the MVEL context. Map<String, Object> vars = new HashMap<String, Object>(); QProduct qProduct = QProduct.product; vars.put("type", qProduct.type); vars.put("prize", qProduct.prize); vars.put("creationDate", qProduct.creationDate); vars.put("_date", new DateFactory()); return (Predicate)MVEL.eval(expression, vars); }
If you want to use an entity property in a filter expression which type is not a primitive type or String, but for instance a java.util.Date object, then you can use a object factory for this. A factory creates an object instance from a string representation.
With the following factory you can construct date objects as part of a filter expression:
import org.joda.time.format.DateTimeFormat;
import org.joda.time.format.DateTimeFormatter;

public class DateFactory {

    public String getName(){
        return "_date";
    }

    public Date valueOf(String value) {
        DateTimeFormatter formatter = DateTimeFormat.forPattern("yyyy-mm-dd");
        return formatter.parseDateTime(value).toDate();
    }
}
You can see in the example code above the _date and valueof strings are added to the allowedMethods set for parsing the expression. Furthermore a DateFactory instance is put to the vars map with the name _date, so it is available when evalutaion the expression by MVEL. You use a date in a filter expression like this:
http://www.myprettyproducts.com/product?filter="type.eq(3).and(creationDate.after(_date.valueof('2012-05-23')))"
Of course you will need to parameterize the above code to make it reusable for different use cases.

Friday, March 30, 2012

Filter expressions in Rest urls leveraging QueryDsl Part I

The other day I was looking for a way to create dynamic filter expressions for use in a Restful webservice url. Clients should be able to pass arbitrary filter expressions to filter the results of a GET operation on a collection resource, which values are retrieved from a database by JPA. This idea came to me while using QueryDsl to construct where clauses in a typesafe manner. It would be nice if we could use QueryDsl-like BooleanExpressions as value of a Url query-parameter, like so:
http://www.myprettyproducts.com/product?filter="type.eq(3).and(prize.lt(500))"
In QueryDsl you use the generated QueryType classes to construct a BooleanExpression object which you can use in the where clause of a select query like this:
QProduct qProduct = QProduct.product;
  JPQLQuery query = new JPAQuery(entityManager).from(qProduct)
  .where(qProduct.type.eq(3).and(qProduct.prize.lt(500));
  return  query.list(qProduct);
Now if I could use qProduct.type.eq(3).and(qProduct.prize.lt(500) in a url and convert this string expression to Java code we are done.
Hey, wait a minute are we opening up the system for code injection? Yes, but there is a solution to this I will cover later on.
So how do we convert the expression string to Java code at runtime? I use MVEL for this part. That's a powerful evaluation language for Java applications, which evaluates a script. Now how it works is that you setup a map of key value pairs for the variables that are expected in the expression string. For the example above this would be: "type", qProduct.type and "prize", qProduct.prize. Code example:
This following code will take any valid expression string using the type and prize attributes. The conditional operators are the ones allowed by the QueryDsl QueryType metadata objects and depend on the attributes Type.
Predicate toProductPredicate(String expressionFilter){
  Map<String, Object> vars = new HashMap<String, Object>();
  vars.put("type", qProduct.type);
  vars.put("prize", qProduct.prize);
  return (Predicate)MVEL.eval(expressionFilter, vars);
}

public Collection<product> findAllProducts(String filterExpression){
  QProduct qProduct = QProduct.product;
  JPQLQuery query = new JPAQuery(entityManager).from(qProduct)
            .where(toProductPredicate(expressionString));
  return query.list(qProduct);
}

How to go about the code injection vulnerability.
You want to restrict the expression to only include terms and constructions that lead to a valid QueryDsl Predicate and nothing more. For this purpose I wrote a custom dynamic Parser using Parboild. Parboild is a socalled PEG (Parsing expression grammar) parser. See all detail in the followup post: Filter expressions in Rest urls leveraging QueryDsl Part 2