Embedded Systems Testing Manifesto

Experience and ISTQB® syllabus led to creation of our Embedded Systems Testing Manifesto

#embeddedsystems, #qa, #testing, #standards, #manifesto, #ISTQB

Embedded software QA and testing is a crucial aspect of developing reliable and safe systems for various domains, such as automotive, aerospace, medical, and industrial. In this blog post, we would like to point out the key areas of embedded software and hardware QA and testing, and how they can help ensure high-quality products and how Embevity handles these issues and which tools we use.
We have gathered significant experience in recent years and have paid for our early years’ ignorance when testing was barely done. And it is well known, that bugs discovered late are very costly. There have been many famous examples of failures, due to undiscovered bugs that have been passed to the final product leading to a disaster:

• The Therac-25 radiation therapy machine: A software defect caused the machine to deliver lethal doses of radiation to patients.
• The Ariane 5 rocket explosion: A software error caused the rocket to veer off course and explode shortly after launch.
• The Mars Climate Orbiter: A failure to convert units from English to metric caused the spacecraft to crash into Mars.
• The Patriot Missile Failure.

So after years of working with clients, we have defined the following elements as our Testing Environment Manifesto, which we also recommend: 

1. To have a well-defined and updated testing strategy with specific methods
2. To use a separate, managed team of testers and QA’s
3. Implement quality assurance procedures which increase the quality of the development and testing output
4. Use standards and testing principles and minimize the risk
5. Automate where it makes sense
6. Manage issues tracking
7. Archiving the test environment
8. Agile methodology

1. Testing strategy definition

Defining a strategy for embedded systems testing is an important step in ensuring the quality and reliability of the software and hardware. A testing strategy should include the following elements: objectives, test specification, test levels, test techniques, tools, test environment, data, cases, test execution, reporting and evaluation. A testing strategy should also consider the specific characteristics and challenges of embedded systems, such as limited resources, real-time constraints, hardware dependencies and safety requirements.

The variety of projects implemented at Embevity means that we approach each project individually. Our QA &  Testing team analyzes each project separately and defines a testing strategy for it. Our testing strategy is constantly verified and updated to become the most optimal and, above all, reliable.

2. Independent testing team

The developer in most cases will be gently for his code, which is not conductive to quality assurance. Independent testers (which in nature should have a critical eye, professional pessimism, curiosity and attention to details) have a perspective different than designers and developers. Please note that the responsibility for quality does not transfer to the testing team at this point – it is still an important element of the developer’s work.

Taught by experience, we will not give the product to the customer before it passes through the hands of the tester and quality control. Our testers will do everything to ensure that the product breaks in their hands, not in the client’s.

3. Implementing procedures boosting code quality

Well-defined requirements, risk analysis, technical concept test plan, specifications, and finally user acceptance tests are the best way to ensure quality in the project. Thanks to this, we know when done is done for our client. Continuous step-by-step project control minimizes the risk of going backwards and wasting time and money. At Embevity, we know that defining requirements, developing analyzes and reports is not a job for an hour or two, and thus – it must involve cost. By going through this procedure, we minimize the risk and cost associated with delivering a defective product to the end customer, regardless of the scale and budget of the project.

4. Standardization and following principles

Developed standards make it easier to understand the testing process, make the testing process complete and not incidental. It is the randomness of actions that exposes us to overlooking an error in the created product; often critical and costly. Standards are a set of rules and guidelines, the implementation of which in the project helps to ensure its high quality. The most popular standards we use are ISO/IEC/IEEE 29119 (parts 1-4), ISO/IEC 25010 and ISO/IEC 20246. All the most important rules are developed by the International Software Testing Qualifications Board (ISTQB®), whose materials and knowledge provided are based, among others, on these standards and provide a solid knowledge base for any tester.
Our testers also use the knowledge gained while obtaining the ISTQB® certification. This shortens the decision-making process and the testing process becomes more efficient.

A perfect example of the content of the materials presented by International Software Testing Qualifications Board (ISTQB®), are seven testing principles, which are general guidelines for all testing – embeded testing also :

a) Testing shows the presence of defects, not their absence – it means, that testing can show that defects exists but cannot prove that they do not exist at all. In other words, testing reduces the probability of defects but even if no defects are found, there is no proof that system or software is fully correct.

b) Exhaustive testing is impossible – There is no possibility to test everything with all preconditions and combinations of inputs. More important than test efforts is the focus on risk analysis, test techniques and priorities.

c) Early testing saves time and money – The static and dynamic test acitivities should be astarted at the earliest possible software development life cycle (so-called ‘shift left). It helps to reduce or eliminate costly changes.

d) Defects cluster together – The prediced defect clusters and the really observed defect clusters during testing are an essential input into a risk analysis used to process of testing.
This is due to the small number of modules usually includes most of the defects detected during pre-release testing or operational failures.

e) Beware of the pesticide paradox – Repeating the same tests over and over leads to
a situation where these tests will no longer find any new defects. The existing tests and test data may be modified or new tests could be necessary. It is compared to pesticides – they are no longer effective after a while. The pesticide paradox is only useful during automated regression – the low number of defects confirms low number of regression defects.

f) Testing is context dependent – Testing is done in different contexts in different ways.
For example – safety-critical industrial control software is tested another that e-commerce apps.

g) Absence-of-errors is a fallacy – There is a belief that testrs will be able to run all possible tests and detect all possible defects, but rules mentioned above shows that it is impossible.
It is a fallacy to expect that detecting and fixing a great number of defects will lead to success of the system.

5. Automate where it makes sense

The first step is to define the subject of testing and the possibility of using automation. Let’s divide the subject into FW/SW and HW. The approach to testing, quality assurance and procedures is the same for both. The difference lies in the choice of the research method.
HW, due to its nature, in most cases will be tested manually using measuring devices (e.g. thermovision, oscilloscope) taking into account the target working environment by means of electromagnetic compatibility tests. SW/FW gives much greater opportunities for test automation because it usually does not require external tools and research. SW/FW iterates more often than HW (whose iteration needs to be recreated), which forces us to automate to some extent.

The next step is the cost and profit analysis; those two words define which test method is good for our project. Test process automation requires more costs in first steps of testing but gives bigger profit when our project has a lot functionalities and releases. In this case, manual testing generates more costs due to time which is needed for their execution. Cost analysis is irreplaceable at this point.

Considering the fact that at Embevity we deal with hardware and software development, conducting such analysis is an integral part of the process of pricing and defining tests.

6. Managing issues tracking

This is a process of recording and managing issues or defects found during development or testing of software. Issue tracking can help monitor the status and progress of issues, assign responsibilities and priorities, facilitate communication and collaboration, and generate reports and metrics.

Each of our projects has a specific place to submit tasks and report bugs. These are, for example, Jira, Redmine, Jazz or other locations indicated by the client. Regardless of the location, the bug report always contains the same information, such as: bug description, bug type (critical/non-critical), test environment version, SF/HW version, link or content of the requirement and test. When an error is detected, it is always clearly communicated so that it does not escape anyone’s attention.

7. Test environment archivization

It is a very important point in testing and the change control process. It allows us to recreate the test environment during debugging, which makes it easier. Archiving also applies to documentation – requirements (client’s and internal), referenced documents for requirements, plans, specifications, all documents that are created during test and development process. Another important thing is to link released SW/HW/FW with the appropriate version of the documents – document named ‘release note’ is a good place to define it.

Orderly document management is a must – this is one of the basic goals of our QA team. It is tedious work, but very important for our client, who sees what he paid for and in what quality the product was delivered. For us, this is a confirmation of the completeness of the testing process.

8. Agile methodology

The use of the Agile methodology greatly facilitates the control of maintaining an appropriate level of quality – especially in iterative and incremental software life cycles, in which new functionalities, changes and refactoring cause frequent code changes. This requires frequent functional testing and regression. Test development can be parallel to software development. The sooner you start testing, the sooner you will eliminate errors. An error detected in the early phase of software development is less costly than the same error discovered in the last phase.

Our team of testers and developers work closely together, and our product manager actively participates in the scheduling process and monitors the agile response to all detected defects. This relationship and daily team meetings make it easier to monitor the situation and determine our place on the delivery roadmap.