Json deserialization with Jackson and Super type tokens

Datatables is a jquery plugin to present tabular information - it can enhance a simple table or can use a AJAX based data and present the information in a tabular form.

Datatables requires the data from the server to follow a specific JSON format for it to be displayed on screen. Consider the case where a list of Member entities is to be displayed, the expected json structure for Members then has to be along these lines:

{
   "aaData":[
      {
         "id":1,
         "first":"one",
         "last":"one",
         "addresses":[

         ],
         "version":0
      },
      {
         "id":2,
         "first":"two",
         "last":"two",
         "addresses":[

         ],
         "version":0
      }
   ],
   "iTotalRecords":100,
   "iTotalDisplayRecords":10,
   "success":true
}

A generic java type can be defined which Jackson can use to generate json of the type shown above, consider the following Java generic type:

package mvcsample.types;
import java.util.List;

public class ListWrapper<T> {
    private List<T> aaData;
    private int iTotalRecords;
    private int iTotalDisplayRecords;
    private  Boolean success;

    public List<T> getAaData() {
  return aaData;
 }
 public void setAaData(List<T> aaData) {
  this.aaData = aaData;
 }
 public int getiTotalRecords() {
  return iTotalRecords;
 }
 public void setiTotalRecords(int iTotalRecords) {
  this.iTotalRecords = iTotalRecords;
 }
 public int getiTotalDisplayRecords() {
  return iTotalDisplayRecords;
 }
 public void setiTotalDisplayRecords(int iTotalDisplayRecords) {
  this.iTotalDisplayRecords = iTotalDisplayRecords;
 }
 public Boolean getSuccess() {
  return success;
 }
 public void setSuccess(Boolean success) {
  this.success = success;
 }   
}

So, with this generic type, to generate a list of Members I would have a parameterized type defined as in this test:

List<Member> members = new ArrayList<>();
members.add(new Member("one", "one"));
members.add(new Member("two", "two"));
ListWrapper<Member> membersWrapper = new ListWrapper<>();
membersWrapper.setAaData(members);
membersWrapper.setiTotalDisplayRecords(10);
membersWrapper.setiTotalRecords(100);
ObjectMapper objectMapper = new ObjectMapper();

StringWriter w = new StringWriter();
objectMapper.writeValue(w, membersWrapper);
String json = w.toString();
System.out.println(json);

And similarly a json for any other type can be generated.

However, what about the other way around, generating the Java type given the json.

Again, consider a case where the json given in the beginning is to be converted to ListWrapper<Member> , I can try a deserialization this way:

ObjectMapper objectMapper = new ObjectMapper();  
ListWrapper<Member> membersUpdated = objectMapper.readValue(json, ListWrapper.class);

Note that above I cannot refer to the class type as ListWrapper<Member>.class, I can only refer to it as ListWrapper.class.

This however will not work and the resulting type will not be a wrapper around Member class, as at runtime Jackson has no idea that it has to generate a ListWrapper<Member>.

The fix is to somehow pass the information about the ListWrapper's type to Jackson and this is where Super type tokens fits in. The article explains how this works in great detail, the essence is that
while type erasure does remove the type information from parameterized instances of generic type, however the type is retained in subclasses of generic classes.

For eg. Consider the following StringList class which derives from ArrayList<String> , it is possible to find that the type parameter of the base class is a String as shown in the test below:

import java.lang.reflect.ParameterizedType;
import java.lang.reflect.Type;
import java.util.ArrayList;

public class StringList extends ArrayList<String>{

 public static void main(String[] args) {
  StringList list = new StringList();
  Type superClassType = list.getClass().getGenericSuperclass();
  ParameterizedType parameterizedType = (ParameterizedType)superClassType;
  System.out.println(parameterizedType.getActualTypeArguments()[0]);
 }
}

This is applicable for a case where a subclass is defined as an anonymous class also this way:

ArrayList<String> list = new ArrayList<String>(){};
Type superClassType = list.getClass().getGenericSuperclass();
ParameterizedType parameterizedType = (ParameterizedType)superClassType;
System.out.println(parameterizedType.getActualTypeArguments()[0]);

This is what is used internally with the Super Type tokens pattern to find the type of the parameterized type. Jackson's com.fasterxml.jackson.core.type.TypeReference abstract class implements this and using this the Jackson deserialization would work this way:

import com.fasterxml.jackson.core.type.TypeReference;

....
ListWrapper<Member> membersWrapper = objectMapper.readValue(json, new TypeReference<ListWrapper<Member>>() {});

ListWrapper<Address> addressWrapper = objectMapper.readValue(json, new TypeReference<ListWrapper<Address>>() {});

This way two different parameterized types can be deserialized given a generic type and a json representation.


References:

Reflecting Generics: http://www.artima.com/weblogs/viewpost.jsp?thread=208860

Neal Gafter's Super type tokens: http://gafter.blogspot.com/2006/12/super-type-tokens.html

Spring Data JPA and pagination

Let us start with the classic JPA way to support pagination.

Consider a simple domain class - A "Member" with attributes first name, last name. To support pagination on a list of members, the JPA way is to support a finder which takes in the offset of the first result(firstResult) and the size of the result(maxResults) to retrieve, this way:

import java.util.List;

import javax.persistence.TypedQuery;

import org.springframework.stereotype.Repository;

import mvcsample.domain.Member;

@Repository
public class JpaMemberDao extends JpaDao<Long, Member> implements MemberDao{

 public JpaMemberDao(){
  super(Member.class);
 }
 @Override
 public List<Member> findAll(int firstResult, int maxResults) {
  TypedQuery<Member> query = this.entityManager.createQuery("select m from Member m", Member.class);
  return query.setFirstResult(firstResult).setMaxResults(maxResults).getResultList();
 }

 @Override
 public Long countMembers() {
  TypedQuery<Long> query = this.entityManager.createQuery("select count(m) from Member m", Long.class);
  return query.getSingleResult();
 }
}

An additional API which returns the count of the records is needed to determine the number of pages for the list of entity, as shown above.

Given this API, two parameters are typically required from the UI:

• the current page being displayed (say "page.page")
• the size of list per page (say "page.size")

The controller will be responsible for transforming these inputs to the one required by the JPA - firstResult and maxResults this way:

@RequestMapping(produces="text/html")
public String list(@RequestParam(defaultValue="1", value="page.page", required=false) Integer page, 
   @RequestParam(defaultValue="10", value="page.size", required=false) Integer size, Model model){
 int firstResult = (page==null)?0:(page-1) * size;
 model.addAttribute("members",this.memberDao.findAll(firstResult, size));
 float nrOfPages = (float)this.memberDao.countMembers()/size;
 int maxPages = (int)( ((nrOfPages>(int)nrOfPages) || nrOfPages==0.0)?nrOfPages+1:nrOfPages);
 model.addAttribute("maxPages", maxPages);
 return "members/list";
}

Given a list as a model attribute and the count of all pages(maxPages above), the list can be transformed to a simple table in a jsp, there is a nice tag library that is packaged with Spring Roo which can be used to present the pagination element in a jsp page, I have included it with the reference.

So this is the approach to pagination using JPA and Spring MVC.

Spring-Data-JPA makes this even simpler, first is the repository interface to support retrieving a paginated list - in its simplest form the repository simply requires extending Spring-Data-JPA interfaces and at runtime generates the proxies which implements the real JPA calls:

import mvcsample.domain.Member;

import org.springframework.data.jpa.repository.JpaRepository;
import org.springframework.stereotype.Repository;

public interface MemberRepository extends JpaRepository<Member, Long>{
 //
}

Given this, the controller method which accesses the repository interface is also very simple:

@RequestMapping(produces="text/html")
public String list(Pageable pageable, Model model){
 Page<Member> members = this.memberRepository.findAll(pageable);
    model.addAttribute("members", members.getContent());
    float nrOfPages = members.getTotalPages();
    model.addAttribute("maxPages", nrOfPages);
 return "members/list";
}

The controller method accepts a parameter called Pageable, this parameter is populated using a Spring MVC HandlerMethodArgumentResolver that looks for request parameters by name "page.page" and "page.size" and converts them into the Pageable argument. This custom HandlerMethodArgumentResolver is registered with Spring MVC this way:

<mvc:annotation-driven>
 <mvc:argument-resolvers>
  <bean class="org.springframework.data.web.PageableArgumentResolver"></bean>
 </mvc:argument-resolvers>
</mvc:annotation-driven> 

the JpaRepository API takes in the pageable argument and returns a page, internally automatically populating the count of pages also which can retrieved from the Page methods.

If the queries need to be explicitly specified then this can be done in a number of ways, one of which is the following:

@Query(value="select m from Member m", countQuery="select count(m) from Member m")
Page<Member> findMembers(Pageable pageable);

One catch which I could see is that that pageable's page number is 0 indexed, whereas the one passed from the UI is 1 indexed, however the PageableArgumentResolver internally handles and converts the 1 indexed UI page parameter to the required 0 indexed value.

Spring Data JPA thus makes it really simple to implement a paginated list page.

I am including a sample project which ties all this together, along with the pagination tag library which makes it simple to show the paginated list.

Reference:
A sample projects which implements a paginated list is available here : https://github.com/bijukunjummen/spring-mvc-test-sample.git

Spring-Data-JPA reference: http://static.springsource.org/spring-data/data-jpa/docs/current/reference/html/

context:component-scan, context:annotation-config and list of beans registered

A <context:component-scan/> custom tag registers the same set of bean definitions as is done by <context:annotation-config/>, apart from its primary responsibility of scanning the java packages and registering bean definitions from the classpath.

The following bean types are registered by either of these custom tags:

Bean	Purpose
ConfigurationClassPostProcessor	A bean factory post processor used for bootstrapping classes annotated with @Configuration
AutowiredAnnotationBeanPostProcessor	A bean post processor which processes fields, setter methods, arbitrary methods annotated with @AutoWired, @Value and supports JSR-330 @Inject annotation
RequiredAnnotationBeanPostProcessor	A bean post processor which enforces that the required bean properties have been set - by looking for fields annotated with @Required annotation
CommonAnnotationBeanPostProcessor	A bean post processor which supports JSR-250 annotations - eg. @Resource
PersistenceAnnotationBeanPostProcessor	A bean post processor that processes @PersistenceUnit and @PersistenceContext annotations for injecting in a EntityManagerFactory and EntityManager respectively.

This is the reason why if component-scan is present, annotation-config can be kept out.

If for some reason this registration of default bean definitions are to be avoided, the way to do that is to specify an additional "annotation-config" attribute in component-scan, this way:

<context:component-scan basePackages=".." annotation-config="false"/>

Reference:
Spring Reference site

Composing Java annotations

The allowed attribute types of a Java annotations are deliberately very restrictive, however some clean composite annotation types are possible with the allowed types.

Consider a sample annotation from the tutorial site:

package annotation;
@interface ClassPreamble {
   String author();
   String[] reviewers();
}

Here the author and reviewers are of String and array types which is in keeping with the allowed types of annotation attributes. The following is a comprehensive list of allowed types(as of Java 7):

• String
• Class
• any parameterized invocation of Class
• an enum type
• an annotation type, do note that cycles are not allowed, the annotated type cannot refer to itself
• an array type whose element type is one of the preceding types.

Now, to make a richer ClassPreable consider two more annotation types defined this way:

package annotation;

public @interface Author {
 String first() default "";
 String last() default "";
}


package annotation;

public @interface Reviewer {
 String first() default "";
 String last() default "";
}

With these, the ClassPreamble can be composed from the richer Author and Reviewer annotation types, this way:

package annotation;
@interface ClassPreamble {
   Author author();
   Reviewer[] reviewers();
}

Now an annotation applied on a class looks like this:

package annotation;

@ClassPreamble(author = @Author(first = "John", last = "Doe")
    , reviewers = {@Reviewer(first = "first1", last = "last1"), @Reviewer(last = "last2") }
)
public class MyClass {
....
}

This is a contrived example just to demonstrate composition of annotations, however this approach is used extensively for real world annotations, for eg, to define a many to many relationship between two JPA entities:

    @ManyToMany
    @JoinTable(name="Employee_Project",
          joinColumns=@JoinColumn(name="Employee_ID"),
          inverseJoinColumns=@JoinColumn(name="Project_ID"))
    private Collection<Project> projects;

Spring test mvc with Spring 3.2RC1

I had an opportunity earlier to try out Spring-test-mvc and was very impressed with how easy it now makes integration testing of the Spring MVC controllers.

Spring-test-mvc is now packaged with Spring-test module as of Spring 3.2RC1 and has made a few changes which is what I wanted to record for myself here:

First is to include the Spring-test-mvc module, the following is the dependency now that it is packaged with spring-test module:

<dependency>
 <groupId>org.springframework</groupId>
 <artifactId>spring-test</artifactId>
 <version>3.2.0.RC1</version>
 <scope>test</scope>
</dependency>

Now on the test side the first change is to include a @WebAppConfiguration annotation:

@WebAppConfiguration
@ContextConfiguration(locations={"file:src/main/webapp/WEB-INF/spring/mvc-config.xml", "classpath:/META-INF/spring/applicationContext.xml", "classpath:/META-INF/spring/applicationContext-jpa.xml"})
@RunWith(SpringJUnit4ClassRunner.class)
public class MemberJsonControllerTest {
 
 @Autowired
 private WebApplicationContext wac;

This new annotation instructs Spring test support to load up a WebApplicationContext - a mock ServletContext is automatically assigned to this WebApplicationContext.

Another change from previous version of Spring-test-mvc is the updated package names, the following are the new static imports:

import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.*;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.*;
import static org.springframework.test.web.servlet.setup.MockMvcBuilders.*;

Now that a WebApplicationContext is available in the scope of the test, the next step is to get hold of MockMvc which will drive all the controller related tests:

private MockMvc mockMvc;

@Before
public void setup() {
 this.mockMvc = webAppContextSetup(this.wac).build();
}

Once a MockMvc instance is available all the controller tests can be driven using this, consider the following test where I have a series of CRUD actions and tests on the results being performed on a controller:

@Test
public void testAControllerFlow() throws Exception {
 this.mockMvc.perform(post("/membersjson").contentType(MediaType.APPLICATION_JSON).content(String.format(createJson, 1,"One","One", 0).getBytes()))
   .andExpect(status().isOk());
 this.mockMvc.perform(post("/membersjson").contentType(MediaType.APPLICATION_JSON).content(String.format(createJson, 2,"Two","Two", 0).getBytes()))
  .andExpect(status().isOk());
 this.mockMvc.perform(post("/membersjson").contentType(MediaType.APPLICATION_JSON).content(String.format(createJson, 3,"Three","Three", 0).getBytes()))
  .andExpect(status().isOk());  
 
 this.mockMvc.perform(get("/membersjson").contentType(MediaType.APPLICATION_JSON))
   .andExpect(status().isOk())
   .andExpect(content().string(containsString(membersJson)));
 
 this.mockMvc.perform(put("/membersjson").contentType(MediaType.APPLICATION_JSON).content(String.format(createJson, 1,"One","OneUpdated", 0).getBytes()))
  .andExpect(status().isOk());
 
 mockMvc.perform(get("/membersjson").contentType(MediaType.APPLICATION_JSON))
  .andExpect(status().isOk())
  .andExpect(jsonPath("$[0].first").value("One"));
 
 mockMvc.perform(get("/membersjson/1").contentType(MediaType.APPLICATION_JSON))
  .andExpect(status().isOk())
  .andExpect(content().string(containsString("OneUpdated")));

}

The tests are very readable thanks to the fluent interfaces and supports matchers on response data based on Hamcrest which I have used above(containsString) and also supports jsonpath for validating json responses.


I have a github project which provides the complete working sample: https://github.com/bijukunjummen/spring-mvc-test-sample.git

Spring Integration Standalone application

Creating a standalone Spring application and by extension a Spring Integration application is very easy. As long as a non-daemon thread is active, the main thread does not terminate, so if the Spring Integration application has some task related or polling related threads active the main task will be alive.

Consider the Spring Integration application that I had outlined in my previous Blog entry:

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xmlns:int="http://www.springframework.org/schema/integration"
 xmlns:int-http="http://www.springframework.org/schema/integration/http"
 xsi:schemaLocation="http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration.xsd
  http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
  http://www.springframework.org/schema/integration/http http://www.springframework.org/schema/integration/http/spring-integration-http.xsd">

 <int:inbound-channel-adapter channel="quakeinfotrigger.channel" expression="''">
  <int:poller fixed-delay="60000"></int:poller>
 </int:inbound-channel-adapter>
 
 <int:channel id="quakeinfo.channel">
  <int:queue capacity="10"/>
 </int:channel>

 <int:channel id="quakeinfotrigger.channel"></int:channel> 

 <int-http:outbound-gateway id="quakerHttpGateway"
     request-channel="quakeinfotrigger.channel"
     url="http://earthquake.usgs.gov/earthquakes/feed/geojson/all/hour"
     http-method="GET"
     expected-response-type="java.lang.String"
     charset="UTF-8"
     reply-timeout="5000"
     reply-channel="quakeinfo.channel">     
 </int-http:outbound-gateway>
 
 <int:logging-channel-adapter id="messageLogger" log-full-message="true" channel="quakeinfo.channel" level="ERROR">
  <int:poller fixed-delay="5000" ></int:poller>
 </int:logging-channel-adapter>

</beans>

The above Spring Integration flow polls a http endpoint every minute and prints the contents to System out. Now to write a standalone application out of this, is as simple as the following:

package standalone;

import org.springframework.context.support.ClassPathXmlApplicationContext;

public class Main {

 public static void main(String[] args) {
  ClassPathXmlApplicationContext applicationContext = new ClassPathXmlApplicationContext("classpath:/httpgateway.xml");
  applicationContext.registerShutdownHook();
 }
}

Initialize the ApplicationContext, and register a shutdown hook for orderly shutdown of the context, that's it! Since the sample flow above spawns threads to poll an http endpoint the application stays active with just the above code.

This sample is available at the following github repo:
https://github.com/bijukunjummen/si-standalone-sample

Polling an http end point using Spring Integration

It is a little non-intuitive if you want to write a flow with Spring Integration which polls an http end point and gathers some content from the http end point for further processing.

Spring Integration provides a couple of ways for integrating with a HTTP endpoint -

1. Http Outbound adapter - to send the messages to an http endpoint
2. Http Outbound gateway - to send messages to an http endpoint and to collect the response as a message

My first instinct to poll the http endpoint was to use a Http Inbound channel adapter, the wrong assumption that I made was that the adapter will be responsible for getting the information from an endpoint - what Http Inbound Gateway actually does is to expose an Http endpoint and wait for requests to come in! , this is why I started by saying that it was a little non-intuitive to me that to poll a URL and collect content from it I will actually have to use a Http Outbound gateway

With this clarified, consider an example where I want to poll the USGS Earth Quake information feed available at this url - http://earthquake.usgs.gov/earthquakes/feed/geojson/all/hour

This is how my sample http Outbound component looks like:

<int:channel id="quakeinfo.channel">
  <int:queue capacity="10"/>
 </int:channel>

 <int:channel id="quakeinfotrigger.channel"></int:channel> 

 <int-http:outbound-gateway id="quakerHttpGateway"
     request-channel="quakeinfotrigger.channel"
     url="http://earthquake.usgs.gov/earthquakes/feed/geojson/all/hour"
     http-method="GET"
     expected-response-type="java.lang.String"
     charset="UTF-8"
     reply-timeout="5000"
     reply-channel="quakeinfo.channel">     
 </int-http:outbound-gateway>

Here the http outbound gateway waits for messages to come into the quakeinfotrigger channel, sends out a GET request to the "http://earthquake.usgs.gov/earthquakes/feed/geojson/all/hour" url, and places the response json string into the "quakeinfo.channel" channel

Testing this is easy:

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration("httpgateway.xml")
public class TestHttpOutboundGateway {
 
 @Autowired @Qualifier("quakeinfo.channel") PollableChannel quakeinfoChannel;
 @Autowired @Qualifier("quakeinfotrigger.channel") MessageChannel quakeinfoTriggerChannel;

 @Test
 public void testHttpOutbound() {
  quakeinfoTriggerChannel.send(MessageBuilder.withPayload("").build());
  Message<?> message = quakeinfoChannel.receive();
  assertThat(message.getPayload(), is(notNullValue()));
 }

}

What I am doing here is getting a reference to the channel which triggers the outbound gateway to send a message to the http endpoint and reference to another channel where the response from the http endpoint is placed. I am triggering the test flow by placing a dummy empty message in the trigger channel and then waiting on message to be available on the response channel and asserting on the contents.

This works cleanly, however my original intent was to write a poller which would trigger polling of this endpoint once every minute or so, to do this what I have to do is essentially place a dummy message into the "quakeinfotrigger.channel" channel every minute and this is easily accomplished using a Spring Integration "poller" and a bit of Spring Expression language:

<int:inbound-channel-adapter channel="quakeinfotrigger.channel" expression="''">
 <int:poller fixed-delay="60000"></int:poller>
</int:inbound-channel-adapter>

Here I have a Spring inbound-channel-adapter triggered attached to a poller , with the poller triggering an empty message every minute.

All this looks a little convoluted but works nicely - here is a gist with a working code

References:

1. Based on a question I had posed at the Spring forum http://forum.springsource.org/showthread.php?130711-Need-help-with-polling-to-a-json-based-HTTP-service