Saturday, December 27, 2014

Spring retry - ways to integrate with your project

If you have a need to implement robust retry logic in your code, a proven way would be to use the spring retry library. My objective here is not to show how to use the spring retry project itself, but in demonstrating different ways that it can be integrated into your codebase.

Consider a service to invoke an external system:

package retry.service;

public interface RemoteCallService {
    String call() throws Exception;
}


Assume that this call can fail and you want the call to be retried thrice with a 2 second delay each time the call fails, so to simulate this behavior I have defined a mock service using Mockito this way, note that this is being returned as a mocked Spring bean:

@Bean
public RemoteCallService remoteCallService() throws Exception {
    RemoteCallService remoteService = mock(RemoteCallService.class);
    when(remoteService.call())
            .thenThrow(new RuntimeException("Remote Exception 1"))
            .thenThrow(new RuntimeException("Remote Exception 2"))
            .thenReturn("Completed");
    return remoteService;
}
So essentially this mocked service fails 2 times and succeeds with the third call.

And this is the test for the retry logic:

public class SpringRetryTests {

    @Autowired
    private RemoteCallService remoteCallService;

    @Test
    public void testRetry() throws Exception {
        String message = this.remoteCallService.call();
        verify(remoteCallService, times(3)).call();
        assertThat(message, is("Completed"));
    }
}

We are ensuring that the service is called 3 times to account for the first two failed calls and the third call which succeeds.

If we were to directly incorporate spring-retry at the point of calling this service, then the code would have looked like this:
@Test
public void testRetry() throws Exception {
    String message = this.retryTemplate.execute(context -> this.remoteCallService.call());
    verify(remoteCallService, times(3)).call();
    assertThat(message, is("Completed"));
}

This is not ideal however, a better way would be where the callers don't have have to be explicitly aware of the fact that there is a retry logic in place.

Given this, the following are the approaches to incorporate Spring-retry logic.

Approach 1: Custom Aspect to incorporate Spring-retry

This approach should be fairly intuitive as the retry logic can be considered a cross cutting concern and a good way to implement a cross cutting concern is using Aspects. An aspect which incorporates the Spring-retry would look something along these lines:

package retry.aspect;

import org.aspectj.lang.ProceedingJoinPoint;
import org.aspectj.lang.annotation.Around;
import org.aspectj.lang.annotation.Aspect;
import org.aspectj.lang.annotation.Pointcut;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.retry.support.RetryTemplate;

@Aspect
public class RetryAspect {

    private static Logger logger = LoggerFactory.getLogger(RetryAspect.class);

    @Autowired
    private RetryTemplate retryTemplate;

    @Pointcut("execution(* retry.service..*(..))")
    public void serviceMethods() {
        //
    }

    @Around("serviceMethods()")
    public Object aroundServiceMethods(ProceedingJoinPoint joinPoint) {
        try {
            return retryTemplate.execute(retryContext -> joinPoint.proceed());
        } catch (Throwable e) {
            throw new RuntimeException(e);
        }
    }
}

This aspect intercepts the remote service call and delegates the call to the retryTemplate. A full working test is here.

Approach 2: Using Spring-retry provided advice

Out of the box Spring-retry project provides an advice which takes care of ensuring that targeted services can be retried. The AOP configuration to weave the advice around the service requires dealing with raw xml as opposed to the previous approach where the aspect can be woven using Spring Java configuration. The xml configuration looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:aop="http://www.springframework.org/schema/aop"
    xsi:schemaLocation="
    http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop.xsd
    http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd">

 <aop:config>
  <aop:pointcut id="transactional"
       expression="execution(* retry.service..*(..))" />
  <aop:advisor pointcut-ref="transactional"
      advice-ref="retryAdvice" order="-1"/>
 </aop:config>

</beans>

The full working test is here.

Approach 3: Declarative retry logic

This is the recommended approach, you will see that the code is far more concise than with the previous two approaches. With this approach, the only thing that needs to be done is to declaratively indicate which methods need to be retried:

package retry.service;

import org.springframework.retry.annotation.Backoff;
import org.springframework.retry.annotation.Retryable;

public interface RemoteCallService {
    @Retryable(maxAttempts = 3, backoff = @Backoff(delay = 2000))
    String call() throws Exception;
}

and a full test which makes use of this declarative retry logic, also available here:

package retry;

import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.retry.annotation.EnableRetry;
import org.springframework.test.context.ContextConfiguration;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;
import retry.service.RemoteCallService;

import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.is;
import static org.mockito.Mockito.*;


@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration
public class SpringRetryDeclarativeTests {

    @Autowired
    private RemoteCallService remoteCallService;

    @Test
    public void testRetry() throws Exception {
        String message = this.remoteCallService.call();
        verify(remoteCallService, times(3)).call();
        assertThat(message, is("Completed"));
    }

    @Configuration
    @EnableRetry
    public static class SpringConfig {

        @Bean
        public RemoteCallService remoteCallService() throws Exception {
            RemoteCallService remoteService = mock(RemoteCallService.class);
            when(remoteService.call())
                    .thenThrow(new RuntimeException("Remote Exception 1"))
                    .thenThrow(new RuntimeException("Remote Exception 2"))
                    .thenReturn("Completed");
            return remoteService;
        }
    }
}

The @EnableRetry annotation activates the processing of @Retryable annotated methods and internally uses logic along the lines of approach 2 without the end user needing to be explicit about it.

I hope this gives you a slightly better taste for how to incorporate Spring-retry in your project. All the code that I have demonstrated here is also available in my github project here: https://github.com/bijukunjummen/test-spring-retry

12 comments:

  1. Hi Biju.

    If I comment this line in any test:

    // .thenThrow(new RuntimeException("Remote Exception 2"))

    and then run all the tests again, they are still green!

    ReplyDelete
    Replies
    1. Wow, thanks, I did not notice it before. Not completely sure why verify call would break. But it does call the method the correct number of times

      Delete
  2. FYI after cloning your project : https://github.com/bijukunjummen/test-spring-retry.git

    Your build is failing with the same error i've been trying to trouble shoot locally:

    Tests in error:
    SpringRetryAdviceTests.testRetry » IllegalStateException: Failed to load ApplicationContext
    SpringRetryDeclarativeTests.testRetry » IllegalStateException: Failed to load ApplicationContext

    Tests run: 4, Failures: 0, Errors: 2, Skipped: 0

    ReplyDelete
    Replies
    1. Haven't checked the project for a while @Noj, will try today and get back. Some dependency may have broken.

      Delete
  3. package com.paychex.main;

    import org.springframework.boot.autoconfigure.SpringBootApplication;
    import org.springframework.retry.annotation.EnableRetry;

    @EnableRetry
    @SpringBootApplication
    public class App {

    public static void main(String[] args) throws Exception {
    // TODO Auto-generated method stub
    String status = SpringRetryWithHystrixService.getStatus();
    System.out.println(status);
    }

    }

    package com.paychex.main;

    import org.springframework.retry.annotation.EnableRetry;
    import org.springframework.retry.annotation.Recover;
    import org.springframework.retry.annotation.Retryable;

    @EnableRetry
    public class SpringRetryWithHystrixService {

    public static int i=0;

    @Retryable(maxAttempts=5)
    public static String getStatus() throws Exception{
    System.out.println(i++);
    throw new NullPointerException();
    }

    @Recover
    public static String getRecoveryStatus(){
    return new CommandHelloWorld().execute();
    }
    }

    This is my code using spring retry. I want it to print attempt count each time it encounters the NullPointer exception. But it attempsts only once and throws Nullpointer exception. It doesn't retry

    ReplyDelete
    Replies
    1. I'm pretty sure you can't retry static methods.

      Delete
  4. Some point to note for Approach 3:
    Retryable is only working when the object is managed. AKA instantiated as @Bean.

    ReplyDelete
  5. A note for Approach 3:
    Retryable is only working when the object is managed. AKA instantiated as @Bean.

    ReplyDelete
  6. Can just use Observable/Flux, especially if any of those libraries (RxJava/project reactor) is already on classpath.

    ReplyDelete
  7. Hi guys,

    First all, thanks for the post, it's incredible. But I have a comment about the third approach. In the example when you use the "verify" method of Mockito, this dont work as well. If you change the times you want to verify the test still passing. For example, in the third approach, if you change the number of times to verify the call of retry method for 1, the test still passing.

    Currently I have using these approach, but with one modification. In the call of the verify method I'm using the AopTestUtils.getTargetObject() method to get the real object and not the proxy Object. With these changes if you change the number of times to call de retry method for 1, the test dosn't pass, and with these we actually could verify if the method was call the number of times we need.

    Here I put the changes to the third example.



    @Test
    public void testRetry() throws Exception {
    String message = this.remoteCallService.call();
    ((RemoteCallService)verify(AopTestUtils.getTargetObject(remoteCallService, times(3)))).call();
    assertThat(message, is("Completed"));
    }


    Regards

    ReplyDelete
  8. What license is this code released under? Can you put a license on your github repo?

    ReplyDelete