Introduction
These ideas stem from recent work to implement new Juju features; work that I have felt was somewhat hindered in velocity due to working with (against) some of the incumbent Juju test suites.
The current state of unit tests in Juju reflects a historic preference of testing behaviour over implementation.
I will lay out where I think we are now, and propose some notions on where we should aim to be and how we might get there.
The Juju team has formidable engineering talent, so I donât presume to be revealing anything unknown here; more to be pro-active about striving for the betterment of the project.
Current State of Play
Most of our testing, and the default mode of testing is âblack boxâ. Test suites reside in packages outside of the ones we are testing and are generally named â{package}_testâ.
This means the API surface we are allowed to manipulate in tests by default are methods/types/variables exported by the package being tested.
This has the following externalities:
- We test in a âlooseâ loop when writing code, tending to modify large swathes of logic before working out how to test them at arms length in the external test suites.
- In turn we write less testable code. We see things like:
a. Methods that are too long.
b. Instantiation of service dependencies inside logic that should know nothing about such specifics.
c. Logic with concrete dependencies instead of indirections. - Wrangling with the express purpose of making code testable like:
a. âexport_testâ files inside the tested package to expose otherwise package-private logic.
b. Package-level variables that exist for the purpose of being âpatched-outâ by the test suite.
c. Esoteric test composition based on the specific wrangling required to test. - Due to (3), intermittent test failures in CI based on hard-to-reason-about test set-ups.
- Development is slower than it might be due to the burdens imposed by working with test logic.
A Better Approach
While nothing can replace integration tests, unit tests are not quite intended to meet the same need. The former obviously verifies that the the end product correctly implements documented behaviour. The latter are more of an aid to development itself, in particular:
- To enable testing in a tight loop so that logic emerges as small, well tested, verifiably correct modules.
- To produce decoupled code so that modifications are easier/faster to execute.
- From (1) and (2) to produce testable logic.
I am not a fan of the dogmatic TDD approach, but my preference is to test in a tighter loop than I currently tend to do when working on Juju.
What I think we should work towards is unit testing inside the tested package by default, with well chosen integration tests in the external packages. This would manifest in:
- Better, faster tests.
- Smaller, modular code.
- Cleaner logic that is not polluted for exposure to external test suites.
- Clean lines of indirection and adherence to the (SOLI)Dependency Inversion Principle
- Better behaved CI infrastructure.
Getting There
In a perfect world, with infinite time we would:
- Create a same-package test suite for all packages.
- Rename the external test files as â{package}_integration_testâ
- Delete all export_test files and move tests relying on them into the new unit test suites.
- Delete integration tests that flex large swathes of logic repeatedly to poke inner units and rewrite those as local unit tests.
But we canât turn the Titanic on a dime, so these are some things I think would make things cleaner in the short term:
- Write new tests as real unit tests, reducing integration tests over time to a well chosen set.
- Write test suites in the âpackage_testâ files and keep them local to the package.
- No new âexport_testâ additions.
- Move patching and âexport_testâ and other test-specific logic inside the package-local suites, gradually deleting âexport_testâ files.
- Use these test suites for both unit and integration tests, so that âpackage_testâ is the sole location for test composition logic.
- Use more factory patterns in the dependency engine/manifolds to reduce concrete service instantiation.
I would love to hear othersâ thoughts on this.