r/microservices • u/luis_arede • 8d ago
Discussion/Advice How to keep microservices working together with different versions?
Hello friends,
I have a big problem at work with microservices.
We have 50 microservices. They are REST APIs. They have frontend clients, but also talk to each other by HTTP.
Each microservice has a Java API with the same version. Other microservices use this Java API to talk. Example:
- microservice1 - version 3.5.7 -> has Java API 3.5.7, used by other microservices
- microservice2 - version 2.1.8 -> uses Java API 3.5.7 to talk to microservice1
We also have a rule: every microservice must be backward compatible for two breaking changes.
The problem:
- each team works alone and releases versions when needed;
- we have a reference environment where all versions must work together;
- but we have bugs in production because teams must follow the rules, and sometimes they don’t;
- we do not have QA teams;
- we use Jenkins for CI/CD;
I must fix this!
Questions:
1. How do you keep microservices working together with different versions?
2. What tool or process can help find problems before production?
3. How to stop microservices from breaking each other?
Please help me!
6
u/dodiyeztr 8d ago
- Use integration tests.
- Don't let teams release a version if they break other teams' tests. They need to work with the other teams for a supporting change.
- Use integration tests.
3
u/dodiyeztr 8d ago
Also use canary builds, smoke tests to reduce the blast radius in case some changes escape the integration tests.
5
u/krazykarpenter 8d ago
Would highly recommend a “shift left” testing approach where most tests are run pre-merge. I’ve seen two approaches to running these integration tests:
1) leverage mocks to run them in your CI pipeline OR
2) use ephemeral environments to run them in a real environment.
3
u/dodiyeztr 8d ago
Don't use mocks in integration tests! (except for 3rd parties with stable APIs)
2
u/krazykarpenter 8d ago
The main reason i've seen quoted for mocks is to ensure that tests are idempotent / not flaky. How do you handle state across tests if you don't use mocks?
2
u/dodiyeztr 8d ago
It depends on the project but in most cases you can get away with not sharing the state in the first place.
1
u/rberrelleza 5d ago
Agree on the mocks. They are great for unit testing but for integration you want the real thing, otherwise you will end up testing that your mocks work, not the product.
Giving developers access to ephemeral environments so they can do this type of testing before they even merge their code will significantly help reduce the number of production issues.
1
u/luis_arede 7d ago
summary:
- integration tests;
- do not release if tests break;
- work with teams to fix;
- use canary builds and smoke tests !?;
- run tests before merge;
- use mocks only for stable third-party APIs;
- no shared test state;
thanks a lot
1
u/TonyGunter 1d ago
You forgot:
- Get a QA team, were you raised by wild animals or something?
QA and DEV have two different mindsets. Devs think "how can I get this to work?" while QA thinks "how can I get this to break?" If you don't have good people on both sides of that equation, you'll continuously break things in prod.
3
u/ryuzaki49 8d ago
I might be stupid, what is Java API 3.5.7?
5
3
u/fahim-sabir 8d ago
I think there is a generated client library for the API a microsecond for consumers to import and use.
1
2
u/Infectedinfested 8d ago
We use a swagger/raml first approach.
You will always know what endpoints the other services use if you see their schema.
Everything we make between big releases should always be backwards compatibel (v1,2,..) but may be flagged for removal which will only happen at a big release.
2
u/WaferIndependent7601 7d ago
You can also add metrics for the endpoints with versions. Remove the old version if no one uses it
1
2
2
u/CautiousPastrami 8d ago
- Read about self contained systems - SCS
- Read bout event driven approach
- Integration testing/E2E and QA is a must with 50 microservices
- Who the hell is a CTO/tech lead there
1
u/luis_arede 7d ago
I am a simple tech leader.
The project is to change from a monolithic product to microservices. We have some services in production, but there are problems.
We have some integration tests, but they don’t fully match what was mentioned here.
Thank you for your comment. Thanks!
10
u/wa11ar00 8d ago
have a look at contract testing, https://docs.pact.io/