Polling an http end point using Spring Integration

It is a little non-intuitive if you want to write a flow with Spring Integration which polls an http end point and gathers some content from the http end point for further processing.

Spring Integration provides a couple of ways for integrating with a HTTP endpoint -

1. Http Outbound adapter - to send the messages to an http endpoint
2. Http Outbound gateway - to send messages to an http endpoint and to collect the response as a message

My first instinct to poll the http endpoint was to use a Http Inbound channel adapter, the wrong assumption that I made was that the adapter will be responsible for getting the information from an endpoint - what Http Inbound Gateway actually does is to expose an Http endpoint and wait for requests to come in! , this is why I started by saying that it was a little non-intuitive to me that to poll a URL and collect content from it I will actually have to use a Http Outbound gateway

With this clarified, consider an example where I want to poll the USGS Earth Quake information feed available at this url - http://earthquake.usgs.gov/earthquakes/feed/geojson/all/hour

This is how my sample http Outbound component looks like:

<int:channel id="quakeinfo.channel">
  <int:queue capacity="10"/>
 </int:channel>

 <int:channel id="quakeinfotrigger.channel"></int:channel> 

 <int-http:outbound-gateway id="quakerHttpGateway"
     request-channel="quakeinfotrigger.channel"
     url="http://earthquake.usgs.gov/earthquakes/feed/geojson/all/hour"
     http-method="GET"
     expected-response-type="java.lang.String"
     charset="UTF-8"
     reply-timeout="5000"
     reply-channel="quakeinfo.channel">     
 </int-http:outbound-gateway>

Here the http outbound gateway waits for messages to come into the quakeinfotrigger channel, sends out a GET request to the "http://earthquake.usgs.gov/earthquakes/feed/geojson/all/hour" url, and places the response json string into the "quakeinfo.channel" channel

Testing this is easy:

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration("httpgateway.xml")
public class TestHttpOutboundGateway {
 
 @Autowired @Qualifier("quakeinfo.channel") PollableChannel quakeinfoChannel;
 @Autowired @Qualifier("quakeinfotrigger.channel") MessageChannel quakeinfoTriggerChannel;

 @Test
 public void testHttpOutbound() {
  quakeinfoTriggerChannel.send(MessageBuilder.withPayload("").build());
  Message<?> message = quakeinfoChannel.receive();
  assertThat(message.getPayload(), is(notNullValue()));
 }

}

What I am doing here is getting a reference to the channel which triggers the outbound gateway to send a message to the http endpoint and reference to another channel where the response from the http endpoint is placed. I am triggering the test flow by placing a dummy empty message in the trigger channel and then waiting on message to be available on the response channel and asserting on the contents.

This works cleanly, however my original intent was to write a poller which would trigger polling of this endpoint once every minute or so, to do this what I have to do is essentially place a dummy message into the "quakeinfotrigger.channel" channel every minute and this is easily accomplished using a Spring Integration "poller" and a bit of Spring Expression language:

<int:inbound-channel-adapter channel="quakeinfotrigger.channel" expression="''">
 <int:poller fixed-delay="60000"></int:poller>
</int:inbound-channel-adapter>

Here I have a Spring inbound-channel-adapter triggered attached to a poller , with the poller triggering an empty message every minute.

All this looks a little convoluted but works nicely - here is a gist with a working code

References:

1. Based on a question I had posed at the Spring forum http://forum.springsource.org/showthread.php?130711-Need-help-with-polling-to-a-json-based-HTTP-service

JSP Expression Language and .class

This is something which I faced when using JSP-EL this week, consider the following JSP-EL statement which displays content based on the type of an object:

<c:choose>
 <c:when test="${container.class.simpleName=='TableContainer'}">
  <table>...</table>
 </c:when>
 <c:when test="${container.class.simpleName=='ListContainer'}">
  <ul>...</ul>
 </c:when>
 <c:otherwise>
  <c:out value="Wrong container Type"/>
 </c:otherwise>
</c:choose>

However if this is run on Tomcat 7.0+, you are likely to see a very unhelpful exception, along these lines:

org.apache.jasper.JasperException: /WEB-INF/tags/formlayout/display/displaycontainer.tag (line: 11, column: 1) "${container.class.simpleName=='TableContainer'}" contains invalid expression(s): javax.el.ELException: Failed to parse the expression [${container.class.simpleName=='TableContainer'}]

The actual cause of the issue is the use of the .class above - The expression language parser in Tomcat 7+ by default performs a check to ensure that all identifiers conform to Java Language specifications, since class(which is actually a shortcut for getClass()) is a java Keyword it fails the check for the name of an identifier and hence the error -

There are multiple fixes possible for this issue, the first fix is simply to use .getClass() instead of just .class -

....
 <c:when test="${container.getClass().simpleName=='TableContainer'}">
...
</c:choose>

The second fix is a more sensible one, don't depend on the type of the class, instead introduce a method in the class to describe its type and avoiding Java identifier issue altogether:

....
<c:when test="${container.type=='table'}">
...

Another fix is to soften the identifier check in Tomcat 7+, this can be done using a system property set at Tomcat startup -

org.apache.el.parser. SKIP_IDENTIFIER_CHECK

and is described in more detail here - http://tomcat.apache.org/tomcat-7.0-doc/config/systemprops.html

Method Parameter Names and Spring

Continuing on the previous blog entry about Constructor and method parameters and Java not retaining the parameter names at runtime - the previous entry was about constructor not retaining the parameter names and the implication of this for Contructor injections in Spring, here I will cover a few more scenarios where parameter names not being retained has implications with Spring:

1. Consider Spring MVC Controller method with a parameter to bind to a request parameter which is passed in:

@RequestMapping(value="/members/find")
 public String getMembersByName(@RequestParam String name){
  ...
  return "list";
 }

Here the parameter "name" has a @RequestParam annotation associated with it, which is in an indication to Spring MVC to bind a request parameter "name" to this method parameter.

Since the parameter name is not retained at runtime, it is likely that an exception will be thrown by Spring:

Request processing failed; nested exception is java.lang.IllegalArgumentException: Name for argument type [java.lang.String] not available, and parameter name i
nformation not found in class file either.

The fix here is simple, to either compile with debug options on which will retain the parameter names at runtime OR a better one is to simply indicate what the expected request parameter name is, as an argument to the @RequestParam annotation:

@RequestMapping(value="/members/find")
public String getMembersByName(@RequestParam("name") String name){
 return "list";
}

2. Along the same lines consider another Spring MVC controller method, this time supporting URI template patterns:

@RequestMapping(value="/members/{id}", method=RequestMethod.GET)
public @ResponseBody Member get(@PathVariable Integer id){
 return this.memberDB.get(id);
}

Here the expectation is that if a request comes in with a uri of /members/20, then the id parameter will get bound with a value of 20, however since at runtime the parameter name of "id" is not retained, the fix like in the previous case is either to compile with debug on, or to explicitly mention in the @PathVariable annotation what the expected pattern name is:

@RequestMapping(value="/members/{id}", method=RequestMethod.GET)
public @ResponseBody Member get(@PathVariable("id") Integer id){

3. A third example is with caching support in Spring with @Cacheable annotation. Consider a sample method annotated with @Cacheable:

@Cacheable(value="default", key="#param1.concat('-').concat(#param2)")
public String cachedMethod(String param1, String param2){
    return "" + new Random().nextInt();
}

Here the key is a Spring-EL expression, which instructs the keygenerator to generate the key by combining the argument of the first parameter of name param1 with argument to the second parameter with name param2. However the problem like before is that these names are not available at runtime.

One of the fixes, as before is to compile with debug symbols turned on. A second fix is to use placeholders to stand in for parameter index - a0 OR p0 for first parameter, a1 OR p1 for second parameter and so on, this way the @Cacheable key will look like this:

@Cacheable(value="default", key="#p0.concat('-').concat(#p1)")
public String cachedMethod(String param1, String param2){
    return "" + new Random().nextInt();
}

So in conclusion, a safe way to use Spring features that depend on method parameter names is to compile with debug on(-g or -g:var option of javac) or by explicitly passing in meta information that indicates what the parameter names are at runtime.

Spring Constructor Injection and parameter names

At runtime, java classes do not retain the name of the constructor or method parameters, unless classes are compiled with debug options on. This has some interesting implications for Spring Constructor Injection:

Consider the following simple class:

package dbg;
public class Person {

 private final String first;
 private final String last;
 private final Address address;

 public Person(String first, String last, Address address){
  this.first = first;
  this.last = last;
  this.address = address;
 }

 public String getFirst() {
  return first;
 }

 public String getLast() {
  return last;
 }

 public Address getAddress() {
  return address;
 }
}

and a sample Spring bean configuration xml file:

<bean name="address1" class="dbg.Address" p:street1="street1" p:street2="street1" p:state="state1"/>
 <bean name="person1" class="dbg.Person" c:address-ref="address1" c:last="Last1" c:first="First1"  ></bean>
 <bean name="person2" class="dbg.Person" c:first="First2" c:address-ref="address1" c:last="Last2"   ></bean>

Here I am using the c namespace for constructor injection. This fails with the exception that the argument types are ambiguous - this is because the first argument is a String and since its runtime representation does not have the argument name present, Spring cannot determine if it should be substituted for the first name or last.

There are a couple of fixes possible for this scenario:

1. To use index based constructor injection, the drawback though is that it is very verbose:

<bean name="person1" class="dbg.Person" >
  <constructor-arg value="First1"></constructor-arg>
  <constructor-arg value="Last1"></constructor-arg>
  <constructor-arg ref="address1"></constructor-arg>
 </bean>
 <bean name="person2" class="dbg.Person" >
  <constructor-arg value="First2"></constructor-arg>
  <constructor-arg value="Last2"></constructor-arg>
  <constructor-arg ref="address1"></constructor-arg>
 </bean>

2. To compile with debug symbols on, this can be done by passing a -g or -g:var flag to the java compiler - this will ensure that the parameter names are preserved in the class file and the original concise bean configuration with c namespace will work.


3. A neat fix is to annotate the constructor with @ConstructorProperties which basically provides the argument names to Spring:

public class Person {

 private final String first;
 private final String last;
 private final Address address;

 @ConstructorProperties({"first","last","address"})
 public Person(String first, String last, Address address){
  this.first = first;
  this.last = last;
  this.address = address;
 }

This works with or without debug options turned on.

4. Probably the best fix of all is to simply use @Configuration to define the beans:

@Configuration
public static class TestConfiguration{
 
 @Bean
 public Address address1(){
  return new Address();
 }
 @Bean
 public Person person1(){
  return new Person("First1", "Last1", address1());
 }
 @Bean
 public Person person2(){
  return new Person("First2", "Last2", address1());
 }
 
}

Spring Collection Merging

Spring collection merging is a feature that I first came across as an answer to a StackOverflow question

It is a way of creating a base collection (list, set, map or properties) and modifying this base collection in other beans, best explained using an example -

Consider a Person class with a field holding a list of addresses:

public class Person {
 private List<Address> addresses;
..
}

Assume that all Person instances have a common set of addresses, which can be specified using an abstract person bean this way:

<bean name="basePerson" class="Person" abstract="true">
 <property name="addresses">
  <list>
   <bean class="Address" p:street="Street1" p:state="State1" p:zip="001"></bean>
   <bean class="Address" p:street="Street2" p:state="State2" p:zip="002"></bean>
   <bean class="Address" p:street="Street3" p:state="State3" p:zip="003"></bean>
  </list>
 </property>
</bean>

Now, Person bean instances with addresses in addition to the addresses from the basePerson can be specified using Collection Merging feature this way - note the "merge=true" attribute of list:

<bean name="person1" class="Person" parent="basePerson">
 <property name="addresses">
  <list merge="true">
   <bean class="Address" p:street="Street4" p:state="State4" p:zip="004"></bean>
  </list>
 </property>
</bean>

Collection merging also works with <set/>, <map/> and <props/>

With Spring 3.1 and higher, a simpler option though could be simply to use @Configuraion and @Bean options, as then the merging can be directly handled with Java code, for eg, an equivalent @Configuration for the above xml bean configuration:

@Configuration
public class CollectionConfig{
 
 private List<Address> baseAddresses(){
  return Lists.newArrayList(new Address("Street1", "State1", "001"), new Address("Street2", "State2", "002"), new Address("Street3", "State3", "003"));
 }
 
 
 @Bean
 public Person person1(){
  Person person1 = new Person();
  person1.setAddresses(baseAddresses());
  person1.getAddresses().add(new Address("Street4", "State4", "004"));
  return person1;
 }
}

Spring Autowire - Subtle behavior difference in xml and @Configuration

There is a subtle difference in the behavior of how autowire behaves with new @Configuration style of Spring bean configuration and using an xml file:

Consider a service into which a Dao is wired in:

public class SampleService {
 @Autowired private SampleDao sampleDao;
 
 public Sample getSample(int id){
  return this.sampleDao.findOne(id);
 }
}

Using an xml, the bean configuration would have been defined this way:

<bean name="sampleDao" class="SampleDaoImpl"/>
 <bean name="sampleService" class="SampleService"/>

and using the new @Configuration style it is defined this way:

@Configuration
public class AnnotationConfig {
 @Bean
 public SampleDao sampleDao(){
  return new SampleDaoImpl();
 }
 
 @Bean
 public SampleService sampleService(){
  return new SampleService();
 }
}

Now consider a program which uses the @Configuration defined bean configuration:

public static void main(String[] args) {
  ApplicationContext ctx = new AnnotationConfigApplicationContext(AnnotationConfig.class);
  SampleService sampleService = ctx.getBean(SampleService.class);
  System.out.println(sampleService.getSample(10));
 }

and a similar one using xml bean configuration:

public static void main(String[] args) {
  ClassPathXmlApplicationContext ctx = new ClassPathXmlApplicationContext("springconfig.xml");
  SampleService sampleService = ctx.getBean(SampleService.class);
  System.out.println(sampleService.getSample(10));
 }

The behavior is that the SampleDao will not get correctly autowired into the SampleService in the case of xml based bean configuration but works correctly for @Configuration based bean configuration.

The reason for the behavior difference is that in the @Configuration case the AutowiredAnnotationBeanPostProcessor responsible for autowiring the fields is registered automatically whereas with XML bean configuration it has to be done explicitly. So the fix is to include AutowiredAnnotationBeanPostProcessor, which can be done in a few different ways, one of which is using:

<context:annotation-config/>

Note: I have deliberately demonstrated this in a main method instead of a unit test - In a unit test with Spring test support, the AutowiredAnnotationBeanPostProcessor is automatically registered by the test context and this behavior difference will not be seen there.

Spring MVC - static resource handling with / servlet-mapping

A servlet mapping of "/" registers a "default" servlet - if a request comes in which matches a mapping in the web.xml file then the request will be handled by that servlet, however if no servlet is found matching a specific request pattern then the request is handled by the "default" servelt.

The "default" servlet is also responsible for handling the requests to the static content in a web application.

A convention for Spring MVC based applications is to register the Spring MVC Front Controller called the DispatcherServlet with a servlet mapping of '/'

    <servlet>
        <servlet-name>spring</servlet-name>
        <servlet-class>org.springframework.web.servlet.DispatcherServlet</servlet-class>
        <init-param>
            <param-name>contextConfigLocation</param-name>
            <param-value>classpath:/META-INF/spring/web/webmvc-config.xml</param-value>
        </init-param>
        <load-on-startup>1</load-on-startup>
    </servlet>


    <servlet-mapping>
        <servlet-name>spring</servlet-name>
        <url-pattern>/</url-pattern>
    </servlet-mapping>

This enables Spring MVC to handle all the requests coming to the web application, except for cases where other specific mappings are available. A good example of a case where a specific mapping may be required is to configure the front controller for Apache CXF to accept webservice requests to /webservices servlet path:

 <servlet>
  <servlet-name>CXFServlet</servlet-name>
  <servlet-class>org.apache.cxf.transport.servlet.CXFServlet</servlet-class>
  <load-on-startup>1</load-on-startup>
 </servlet>
 
 
 <servlet-mapping>
  <servlet-name>CXFServlet</servlet-name>
  <url-pattern>/webservices/*</url-pattern>
 </servlet-mapping>

Given that it is the "default" servlet which handles the request for the static resources like images, and the default servlet with Spring MVC is the DispatcherServlet, there are a few recommendations to serve out static content from Spring MVC based web applications:

1. Using mvc:resources -
For eg.

<mvc:resources mapping="/resources/**" location="/public-resources/"/>

This enables serving any static content whose location can be specified as a Spring Resource

2. Using mvc:default-servlet-handler:

 <mvc:default-servlet-handler />

This provides a way to serve out the static content from the root of the web application even though the Dispatcher Servlet is registered at /, the details of how Spring does this is available at the Spring documentation site here - in brief the responsibility is delegated to the containers default servlet.