A little more on Spring tests - good practices

Mateusz Chrzonstowski,

Damian Kaczmarczyk

December 11, 2023

Within Paramount+, we recently improved our integration tests and their execution time, especially on CI.

This article provides hints on how to define good level of tests, how to make tests more readable, and how to use some underlying mechanisms from Spring for easier maintenance.

In the second article, we’ll show what optimizations we did for tests and what you might check in your app.

Challenging the testing pyramid

You probably have heard of a testing pyramid, visualizing unit tests as the base of the software, with integration and E2E tests applied on lower and lower scales.

As in architecture we don’t follow pyramidal shapes anymore, it’s not a surprise the software testing pyramid also got challenged. One of the articles worth providing here is Testing of Microservices by Spotify. There, they propose the following “honeycomb” (a.k.a. “diamond”) shape, where the majority of tests are the integration ones.

Standard pyramid and the honeycomb

The above approach makes sense, but there is also another view on this, where people question rather the “unit” part from the name “unit tests” and define the unit more like a component, e.g. from C4 architecture model. A component can correspond with Java’s package itself and to see how to divide the app into mid-sized building blocks and how to test them not to fix yourself on implementation details, but on behaviors, we recommend checking the following talks from Jakub Nabrdalik:

  1. Keep IT clean: mid-sized building blocks and hexagonal architecture
  2. Improving your Test Driven Development in 45 minutes

The need of integration tests

As we already know it’s worth thinking about bigger units and integration tests themselves, let’s define where the border between tests is. Spring doc on integration testing states:

[Integration Testing] lets you test the correct wiring of your Spring IoC container contexts

In other words, each time you start Spring context, you have an integration test as Spring is responsible for a proper “integration” of your dependencies, even if they don’t touch IO.

With such a definition, we can immediately see integration tests can help us to spot the issues un-spottable even with good component tests, e.g. logic hidden in

  1. @HystrixCommand
  2. @Transactional
  3. @Cacheable
  4. @ConditionalOn...
  5. Filters
  6. Interceptors
  7. @PreAuthorize

Basically, with all Aspect-Oriented Programming, called from time to time “Spring magic”, requiring Spring context and beans management.

Here, it’s worth mentioning, though, that Spring itself provides many tools for testing and the logic coming from @ConditionalOn... can be tested in an unit-like manner (without a real Spring context), just by using Spring’s ApplicationContextRunner.

Batteries included - what tools are already there

ApplicationContextRunner is a nice thing, but what else comes when including spring-boot-starter-test?

  1. spring-boot-test, spring-test - helpers from Spring, e.g. mentioned ApplicationContextRunner, SpringExtension
  2. JUnit (Platform + Jupiter) - tests runner and framework
  3. Mockito - a tool for mocking. BTW it’s worth checking MockitoExtension and BDDMockito which are already there
  4. AssertJ, Hamcrest - fluent assertions, so you don’t need to guess anymore order of parameters if it’s assertEquals(expected, actual) or assertEquals(actual, expected) 😉 it’ll be just assertThat(actual).isEqualTo(expected) or even then(actual).isEqualTo(expected) with BDDAssertions which are already there
  5. Things helping to test JSON, XML and others (worth to mention BasicJsonTester for JSON and OutputCaptureExtension for checking log messages)
  6. Awaitility for active waiting in your tests

Annotations, meta-annotations and their quirks

Both Spring and JUnit interpret annotation put on annotation as if it was put directly on the class. This means whenever you see

@SpringBootTest
@ExtendWith(SpringExtension.class)
class IntegrationTest {
    /* ... */
}

you can safely remove @ExtendWith(SpringExtension.class) as @SpringBootTest itself already adds it (and more):

@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Inherited
@BootstrapWith(SpringBootTestContextBootstrapper.class)
@ExtendWith(SpringExtension.class)
public @interface SpringBootTest {
    /* ... */
}

It gets even more interesting when you check other annotations, e.g.

@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Inherited
@BootstrapWith(DataJpaTestContextBootstrapper.class)
@ExtendWith(SpringExtension.class)
@OverrideAutoConfiguration(enabled = false)
@TypeExcludeFilters(DataJpaTypeExcludeFilter.class)
@Transactional
@AutoConfigureCache
@AutoConfigureDataJpa
@AutoConfigureTestDatabase
@AutoConfigureTestEntityManager
@ImportAutoConfiguration
public @interface DataJpaTest {
    /* ... */
}

JavaDoc hints there, it’s enough to add @AutoConfigureTestDatabase and a DB dependency like H2 to make tests establishing H2 connection with proper Spring properties. No need to set up H2 on your own for tests.

And @Transactional in integration tests is even more powerful. JavaDoc of DataJpaTest says tests marked like that

are transactional and roll back at the end of each test

It means we can get a good level of isolation out of the box - each test runs within its own transaction and doesn’t commit anything. How often did you see some DbCleaner used to remove everything from DB at the end of the test? Using @Transactional could simplify this.

On the other hand, “with great power comes great responsibility”, and JPA with transactions might lead to some problems, described well in the article Spring pitfalls: transactional tests considered harmful. Even though it’s from 2011, most things are still relevant (perhaps just Scala part didn’t age well 😉). But we can ask here if JPA is really needed. Maybe Spring Data JDBC alone or other light solutions are enough?

Lastly, you might often see RestAssured or other additional dependencies for testing your app through HTTP APIs. Instead, you might consider simple @AutoConfigureMockMvc and MockMvc introduced directly by Spring with a fluent API, not worse than in other solutions (but already well integrated and tested by Spring team).

Sharing the setup

Coming up with your own set of annotations you want to use, and knowing both Spring and JUnit understand annotations on annotations, consider introducing your custom meta-annotation, e.g.

@Tag("integration")
@Transactional
@AutoConfigureMockMvc
@SpringBootTest
@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Inherited
public @interface IntegrationTest { }

Then, you can easily reuse the setup across all the test classes:

@IntegrationTest
class LimitingControllerIntTest {
    /* ... */
}

Having a single source of setup is helpful as it’s easy to start many different Spring contexts in integration tests and each comes with its startup and memory use costs. A single point of configuration puts a focus on the common part first, and people might rather work on it, extending it further, instead of introducing a different set of annotations on different test classes.

We’ll cover more on Spring contexts and context caching in the next article.

BTW, with base classes or interfaces you might achieve same benefits as with meta-annotations, but it’s a different discussion, touching composition over inheritance topic among others.

Tests as addons and enablers

Single point of setup and good choice of helpers is desired, but one might still create configurations in production code like

@Configuration
class DbConfigurationBeans {
    @Bean
    @Profile("!test")
    CarouselRepository carouselRepository(DbClient client) {
        return new SpannerCarouselRepository(client);
    }

    // ...
}

This works, but might be considered a code smell. Not only production code is affected by tests (while tests should work as enabler for production code changes, not limiting your production code freedom), but also such things don’t scale well. It’s a matter of time you’ll need another profile, e.g. for “contract tests” where some things should be taken as in prod, some as in current test profile. Then, another profile and another, and soon you might end up with a bunch of configuring annotations or complex expressions used inside @Profile or @ConditionalOn... annotations, which is actually “stringly” typed programming, where errors get discovered at runtime, not at compile-time. Alternatively, you might end up with dozens of profiles and corresponding properties/YAML files, which is also hard to maintain.

Creating and combining profiles is a good topic for a dedicated article. Still, you might consider “feature-oriented” profiles rather than mixing infrastructure, environment, app mode, and other things within the same flat file. Perhaps localstack and aws (with URLs and access details) are good profiles as add-ons to kinesis (with topics), and prodor dev can just include localstack+kinesis or aws+kinesis. Many things can also be simplified by a good naming convention and environment variables, e.g.

base.solr.url: http://solr-cache/${ENVIRONMENT:local}/blog

And instead of a bunch of @Profile annotations, consider dedicated testing properties modifying the production setup. E.g. don’t configure conditionally in production code, but turn something off when testing:

// just in tests
@Configuration(proxyBeanMethods = false)
@ConditionalOnProperty(name = "app.metrics.type", havingValue = "in-mem") // unknown property outside tests
@EnableAutoConfiguration(exclude = PrometheusMetricsExportAutoConfiguration.class) // off!
class PrometheusTestConfiguration {

    @Bean
    CollectorRegistry collectorRegistry() {
        return new CollectorRegistry();
    }
}

Sometimes, you might also override your production beans in tests:

@Bean
@Primary
SpannerOptions spannerEmulatorOptions(
        SpannerOptions original,
        @Value("${spanner.emulator-host}") String spannerHost) {
    return original.toBuilder()
            .setNumChannels(1)
            .setSessionPoolOption(SessionPoolOptions.newBuilder().setMaxSessions(1).build())
            .setEmulatorHost(spannerHost).build();
}

Last words

Good test setup and hygiene are essential, but tests might get out of hand even with a single setup source and start more Spring contexts than needed. In the following article, we’ll cover our journey to put tests back on track and improve their speed.