ATLAS Releasestrategy 3 Qualityassuranceplan

Andi Lamprecht ·2026-05-11· 3 min read· Accepted

ReleaseStrategy - Quality Assurance

Quality Levels

To maintain the highest standards and ensure the reliability of our products, Atlas have established a comprehensive quality assurance framework with four distinct quality levels.

Level 1 - Base On-commit checks

The first level involves base on-commit checks, which are executed with every commit. Those checks are meant to catch and address any issues at the earliest stage of development.

This test level gives an answer to the questions:

• Is the code maintainable?
• Do the building blocks of the system, such as small functions and methods, behave as expected?

Level 2 - Integration Tests

The second level involves integration tests, comprehensively assessing interactions between various project components to ensure seamless functionality. Due to tests complexity and execution time, those tests are meant to run nightly.

This test level gives an answer to the questions:

• Are the integrations between the services works as we expect them to work?
• Are the spark scripts as a whole produce expected results?

Level 3 - System tests

At the third level, nightly system tests evaluate the overall system behavior and performance, simulating real-world scenarios to expose any potential weaknesses. Those tests could be considered as subset of integration tests because it tests the orchestration layer, but in this case it tests end-to-end Airflow dag.

This test level gives an answer to the question:

• Is the system handling correctly the data we expect it to handle?

Level 4 - Data Quality tests

The most important tests level, it could be compared to end-to-end tests in web-dev. Those tests focuses on the constant vigilance of data integrity and accuracy. These tests serve as a solid assurance for Atlas consumers, confirming that the data adheres to the contract’s specifications. Given the continuous flow of data through the system, it becomes imperative to execute these tests at every environmental level, ensuring the reliability of our data products.

This test level gives an answer to the questions:

• Is the data incoming from external sources aligned with our expectations for its format and quality?
• Is our system design to support the data incoming from external sources?
• Is the system generating data in compliance with the agreed-upon data contract?

Test strategy for each environment

1. Dev

The dev environment provides rapid feedback for the developers therefore all of the tests levels are mandatory to be executed here. Although, because it is constantly used by the developers there is a risk that at some point it might slightly differ from staging/production environment which would lead to false-positives test results.

Required tests levels:

• Level 1 - Base On-commit checks
• Level 2 - Integration Tests
• Level 3 - System tests
• Level 4 - Data Quality tests

1. Staging

The staging environment closely replicates the production configuration. With restricted developer access, test outcomes in this environment should ideally match those in production. As it is deployed from the same code branch as the development environment, there is no need for Level 1 tests here.

Required tests levels:

• Level 2 - Integration Tests
• Level 3 - System tests
• Level 4 - Data Quality tests

1. Production

The environment exposed to Atlas consumers. The Highest level of quality must be assured here. Since it has the same configuration as the staging environment the code-quality related tests could be skipped on this level. On the other hand since the data flowing through the system is constantly changing the frequent data quality tests are mandatory to inform both the Atlas and it’s consumers about potential violations of the data expectations.