Code coverage is a metric used to measure the extent to which the source code of a software application has been executed or covered by test cases during testing. It provides insights into how thoroughly the software has been tested and helps identify areas of code that have not been exercised during testing.
In the automotive industry, code coverage is an important aspect of software verification and validation, especially in safety-critical systems. It aligns with relevant standards such as ISO 26262 and IEC 61508, which specify requirements for functional safety in road vehicles and general safety in electrical/electronic/programmable electronic safety-related systems, respectively.
Types of Code Coverage in the Automotive Industry (aligned with ISO 26262 and IEC 61508):
Statement Coverage (SC):
Statement coverage measures the percentage of executable code statements that have been executed during testing.
It helps ensure that each line of code is exercised at least once by the test cases.
This type of coverage is considered a basic measure of code coverage.
Branch Coverage (BC):
Branch coverage, also known as decision coverage, measures the percentage of decision points (branches) in the code that have been exercised during testing.
It ensures that both true and false branches of conditional statements (if-else, switch-case) are tested.
Branch coverage provides more thorough testing than statement coverage as it verifies different code paths.
Function Coverage (FC):
Function coverage measures the percentage of functions or subroutines that have been called and executed during testing.
It ensures that all functions in the software have been exercised at least once.
Condition Coverage (CC):
Condition coverage measures the percentage of Boolean conditions within decision points that have been exercised during testing.
It ensures that all possible combinations of conditions (true/false) within a decision have been tested.
Modified Condition/Decision Coverage (MC/DC):
MC/DC is a stronger form of condition coverage that focuses on the independent testing of each condition within a decision.
It requires that each condition is tested and shown to affect the decision outcome independently.
Multiple Condition Coverage (MCC):
MCC measures the coverage of all possible combinations of conditions in a decision.
It ensures that different combinations of conditions are tested.
In the automotive industry, achieving a sufficient level of code coverage, especially in safety-critical systems, is crucial to demonstrate that the software has been thoroughly tested and that critical functionalities meet the required safety standards. The specific code coverage targets are determined based on the safety integrity level (ASIL) of the system and the requirements of relevant standards like ISO 26262 and IEC 61508.
code coverage targets
The percentage of code coverage expected for different Automotive Safety Integrity Level (ASIL) levels can vary based on industry standards, project requirements, and the specific safety-critical functionalities of the software. ASIL is a classification system defined in the ISO 26262 standard for functional safety of road vehicles. It classifies safety requirements into four levels: ASIL A, ASIL B, ASIL C, and ASIL D, with ASIL D representing the highest level of safety integrity.
It's important to note that ISO 26262 does not prescribe specific code coverage percentages for each ASIL level. Instead, it emphasizes a goal-oriented approach, where the extent of code coverage is determined based on the safety requirements and risks associated with the specific application.
Generally, higher ASIL levels require more extensive and thorough testing, which often results in higher code coverage targets. As the ASIL level increases, the software's safety integrity becomes more critical, and the testing efforts, including code coverage analysis, become more stringent.
Here's a rough guideline for the expected code coverage for different ASIL levels:
ASIL A:
ASIL A represents the lowest safety integrity level, with the least stringent safety requirements.
Code coverage targets for ASIL A may typically be in the range of 50% to 70%.
Basic testing and coverage metrics are sufficient to meet the safety objectives.
ASIL B:
ASIL B requires higher safety integrity than ASIL A, with a moderate increase in the level of testing rigor.
Code coverage targets for ASIL B may typically be in the range of 70% to 80%.
More comprehensive testing and higher code coverage are required to meet the safety goals.
ASIL C:
ASIL C demands a significant increase in safety integrity, with greater emphasis on thorough testing and verification.
Code coverage targets for ASIL C may typically be in the range of 80% to 90%.
Extensive testing and code coverage are necessary to demonstrate compliance with safety requirements.
ASIL D:
ASIL D represents the highest safety integrity level, requiring the most stringent safety measures.
Code coverage targets for ASIL D may typically be in the range of 90% to 100%.
Nearly complete or full code coverage is expected to demonstrate a high level of confidence in the software's safety.
Again, these are general guidelines, and the specific code coverage targets for each ASIL level may vary depending on project-specific factors, system complexity, and safety goals. Automotive manufacturers and development teams should establish code coverage goals based on their safety analysis and in alignment with the relevant safety standards and regulatory requirements.
Requirement impact on code coverage
Functional and non-functional requirements both play important roles in determining the scope and extent of code coverage in the automotive industry. Let's explore how each type of requirement contributes to code coverage:
Functional Requirements:
Functional requirements define what the automotive software or system is supposed to do. They describe the specific functionalities and features that the software must provide to meet user needs and expectations.
Code coverage is directly influenced by functional requirements. Test cases are designed to verify that the software functions as intended, and each functional requirement typically corresponds to one or more test cases.
Test cases based on functional requirements ensure that all specified functionalities are exercised during testing, leading to higher code coverage of the relevant parts of the software.
Functional requirements guide the creation of positive test scenarios (testing expected behaviors) and negative test scenarios (testing error and exception handling), both of which contribute to code coverage.
Non-Functional Requirements:
Non-functional requirements define the qualities and characteristics that the automotive software must possess, beyond its basic functionality. These requirements address aspects such as performance, safety, security, reliability, and usability.
Code coverage is influenced by non-functional requirements, especially those related to safety and security.
Safety-related requirements, as defined in standards like ISO 26262, may require higher code coverage in critical parts of the software. This ensures that safety-critical functionalities are thoroughly tested and verified.
Security requirements might necessitate additional testing and code coverage to ensure that potential vulnerabilities and risks are minimized, and cybersecurity measures are in place.
Performance-related requirements may influence the scope and extent of performance testing, which, in turn, affects code coverage in scenarios that assess the software's behavior under different load conditions.
Overall, both functional and non-functional requirements have a significant impact on code coverage in the automotive industry. Functional requirements guide the design of test cases that exercise the intended functionalities, leading to higher coverage of the functional aspects of the software. On the other hand, non-functional requirements, particularly those related to safety, security, and performance, influence the extent of testing and verification activities required to meet the specific quality and safety objectives. Together, these requirements help ensure that the software is thoroughly tested, and its behavior aligns with the intended functionality and industry standards, resulting in a more reliable and robust automotive system.
What do ASPICE (Automotive SPICE), ISO 26262, and ISO 21434 says about code coverage?
ASPICE (Automotive SPICE), ISO 26262, and ISO 21434 are standards and guidelines that are relevant to the automotive industry and address different aspects of software development and safety.
Let's see how each of these standards addresses code coverage:
ASPICE (Automotive SPICE):
ASPICE is a process assessment model specifically designed for the automotive industry to evaluate the development processes of automotive software and electronic systems.
ASPICE does not explicitly define code coverage requirements. Instead, it focuses on the process maturity, effectiveness, and efficiency of the development processes, including testing and verification activities.
Code coverage is typically a part of the testing activities performed during the software development process to ensure adequate verification and validation.
ISO 26262:
ISO26262 is an international standard for functional safety in road vehicles. It outlines the requirements and processes for achieving functional safety in automotive electrical and electronic systems.
ISO26262 requires a well-defined software development process, including verification and validation activities.
Part6 of ISO 26262 specifically addresses the software development process, including software unit testing and integration testing. The standard requires that the software testing activities demonstrate a sufficient level of code coverage.
ISO26262 does not prescribe a specific code coverage percentage but emphasizes the need for a systematic approach to testing and achieving an appropriate level of coverage based on the safety requirements of the system.
ISO 21434:
ISO21434 is a new standard that focuses on cybersecurity for road vehicles. It provides guidelines for assessing and managing cybersecurity risks during the development and production of automotive systems.
ISO21434 does not have specific requirements for code coverage. Instead, it addresses cybersecurity risk management and the implementation of cybersecurity measures to mitigate potential threats.
Like ASPICE and ISO 26262, ISO 21434 recognizes the importance of testing and validation activities to ensure the security of the software, but it does not dictate specific code coverage metrics.
In summary, while ASPICE, ISO 26262, and ISO 21434 are crucial standards in the automotive industry, they do not provide specific code coverage targets or metrics. However, all these standards emphasize the importance of robust testing, verification, and validation activities, which typically include code coverage analysis, to ensure the safety, quality, and security of automotive software systems. The specific code coverage goals and metrics are typically determined by individual organizations based on their safety and quality objectives and the requirements of their specific projects.
Why code coverage is important ?
Code coverage is essential in the automotive industry, especially in safety-critical systems, for several reasons:
Safety Assurance: In the automotive industry, safety is of paramount importance. Code coverage analysis helps ensure that critical parts of the software have been thoroughly tested, reducing the risk of undetected defects that could lead to potential safety hazards on the road.
Compliance with Standards: Automotive software development is governed by various industry standards, such as ISO 26262. These standards mandate a rigorous development process, including code coverage analysis, to demonstrate that safety-critical software meets the required safety integrity levels (ASIL).
Verification and Validation: Code coverage provides a quantitative measure of how much of the software has been exercised during testing. It helps verify that the testing effort is sufficient to validate the software's correctness and compliance with specifications.
Risk Management: Code coverage identifies areas of the software that have not been tested or have low coverage, allowing development teams to focus their testing efforts on high-risk areas or critical functionalities.
Defect Detection: By analyzing code coverage data, developers can identify untested or under-tested code paths, leading to the detection of potential defects that might otherwise go unnoticed.
Quality Improvement: High code coverage is an indicator of well-tested software and is associated with higher software quality and reliability.
Continuous Integration and Testing: Code coverage is an essential metric for assessing the effectiveness of continuous integration and continuous testing practices. It helps ensure that each code change is thoroughly tested before integration into the software.
Customer Confidence: Demonstrating high code coverage in safety-critical systems enhances customer confidence in the reliability and safety of the automotive software.
Regulatory Compliance: Automotive manufacturers must comply with various governmental regulations, and code coverage is often a requirement to demonstrate software safety and compliance.
Fault Localization: In case of software failures or defects, code coverage data can aid in localizing the root cause, making debugging and issue resolution more efficient.
Overall, code coverage plays a crucial role in the development and validation of automotive software, providing an objective measure of the testing effort and ensuring that safety-critical systems are thoroughly tested and meet the required safety standards. It is a key component of the broader quality assurance and safety validation processes in the automotive industry.
Is 100% code coverage possible?
In the automotive industry, achieving 100% code coverage can be extremely challenging and, in some cases, impractical. Code coverage refers to the percentage of code that has been exercised or executed during testing. While striving for high code coverage is important to ensure software reliability and safety, there are several standard reasons why 100% code coverage may not be achievable or necessary:
Hardware Constraints: In automotive systems, some code may be designed to handle extremely rare scenarios or conditions that are difficult to replicate in a testing environment. These scenarios might require specific hardware configurations or real-world conditions that are hard to simulate comprehensively.
Complexity of Interactions: Modern automotive systems are highly complex and interconnected. Achieving full coverage of all possible interactions between different components can be impractical due to the sheer number of possible scenarios.
Fault Injection Testing: In safety-critical systems like automotive control units, it is often necessary to perform fault injection testing to assess how the software behaves under abnormal or erroneous conditions. This can lead to deliberately triggering parts of the code that would not be exercised during normal operation.
Legacy Code and Third-party Components: Automotive systems may rely on legacy code or third-party components whose source code cannot be modified or accessed. This can limit the ability to achieve complete code coverage for those portions of the software.
Timing and Resource Constraints: Real-time constraints in automotive systems can make it challenging to collect coverage data in certain scenarios, especially during high-load or critical timing situations.
Cost and Time Constraints: Achieving 100% code coverage can be time-consuming and costly. In safety-critical systems, extensive testing and validation are required, and balancing this effort with project timelines and budgets is a key consideration.
Focus on Risk-Based Testing: Rather than aiming for 100% coverage across all code, the focus might be on risk-based testing, prioritizing testing efforts on critical and safety-critical parts of the software.
It's important to note that while 100% code coverage might not be practically achievable, automotive software developers and testers place a strong emphasis on rigorous testing, verification, and validation processes to ensure the safety and reliability of their products. They follow industry standards, such as ISO 26262, to manage risk, perform thorough testing, and establish a level of code coverage that meets the safety requirements for the specific application.
Achieving maximum coverage
Achieving maximum code coverage in automotive industry software requires a comprehensive testing approach that covers various types of test cases and scenarios. Here are some essential types of test cases and scenarios to consider:
Unit Testing: Test individual components or units of code in isolation to ensure their correctness. This involves testing functions, methods, and modules to achieve high coverage at the lowest level of the software.
Integration Testing: Test the interaction between different software modules or components to identify any issues that may arise when they are combined.
Functional Testing: Verify that the software functions as expected according to the specified requirements. Write test cases that cover all functional aspects of the software.
Boundary Value Analysis (BVA): Test scenarios using boundary values of input parameters to ensure that the software behaves correctly at the edges of acceptable ranges.
Equivalence Partitioning: Divide the input domain into equivalent classes and select representative test cases from each class. This technique helps to reduce redundancy in testing and achieve better coverage.
State Transition Testing: For software with state-based behavior, create test cases to cover all possible state transitions and ensure the software behaves correctly in each state.
Negative Testing: Test scenarios where the software should reject invalid inputs or handle error conditions appropriately.
Performance Testing: Evaluate the software's performance under different conditions, such as load testing, stress testing, and resource usage analysis.
Security Testing: Identify and address security vulnerabilities in the software through penetration testing, vulnerability assessments, and access control testing.
Safety-Critical Testing: In the automotive industry, safety is paramount. Write test cases to verify that safety-critical features and functions behave correctly and adhere to safety standards like ISO 26262.
Real-World Scenario Testing: Develop test cases that simulate real-world driving conditions and user scenarios to assess the software's behavior in practical situations.
Regression Testing: After making changes or adding new features, perform regression testing to ensure that existing functionality remains unaffected.
Fault Injection Testing: Deliberately introduce faults or errors to assess the software's robustness and how it handles unexpected situations.
Environmental Testing: Evaluate the software's performance under different environmental conditions, such as temperature extremes or humidity.
Communication Testing: Test communication interfaces and protocols used in the automotive system to ensure data integrity and correct handling of messages.
By incorporating these diverse types of test cases and scenarios into the testing process, automotive software developers can achieve maximum code coverage while ensuring the software's reliability, safety, and adherence to industry standards and requirements.
Requirement vs Test case vs Code coverage in automotive industry
In the automotive industry, Requirement, Test Case, and Code Coverage are interconnected aspects of the software development and testing process. Each plays a crucial role in ensuring the safety, reliability, and compliance of automotive software systems. Let's explore each concept:
Requirement:
Requirements define what the automotive software or system is intended to accomplish. They are specific, measurable, and testable statements that capture the functionalities, features, and performance criteria of the software.
In the automotive industry, requirements are crucial for ensuring that the software meets user needs, safety standards (e.g., ISO 26262), and other regulatory requirements.
Requirements serve as the foundation for the entire development process, guiding the design, implementation, testing, and validation of the software.
Test Case:
A Test Case is a detailed specification that outlines the conditions, inputs, expected outcomes, and execution steps for testing a specific aspect or functionality of the software.
In the automotive industry, test cases are designed to verify that the software behaves as intended and satisfies the specified requirements.
Test cases are developed based on the requirements and cover various scenarios to ensure comprehensive testing of the software.
Code Coverage:
Code Coverage is a metric that quantifies the extent to which the source code of the software has been executed or exercised during testing.
In the automotive industry, code coverage analysis measures the percentage of code (statements, branches, functions, etc.) that has been executed by the test cases.
Code coverage provides insights into the thoroughness of the testing effort and helps identify untested or under-tested parts of the code.
Relation between Requirement, Test Case, and Code Coverage:
Requirement to Test Case:
Test cases are derived from requirements. Each requirement specifies a specific behavior or functionality that needs to be tested, and one or more test cases are developed to verify that requirement.
Test cases are designed to cover positive scenarios (testing expected behavior) and negative scenarios (testing error and exception handling) to ensure comprehensive validation of the requirements.
Test Case to Code Coverage:
Test cases, when executed, exercise parts of the source code. Code coverage analysis measures which portions of the code have been executed or exercised during testing.
The goal of code coverage analysis is to ensure that the test cases adequately exercise the code to meet the required code coverage targets based on safety and quality objectives.
Requirement to Code Coverage:
The ultimate objective of code coverage analysis is to ensure that the code has been thoroughly tested to verify that it meets the specified requirements.
By measuring code coverage against the requirements, development teams can assess the effectiveness of their testing effort and identify areas that require additional testing or validation.
In summary, requirements drive the development of test cases, and the execution of these test cases determines the code coverage. Code coverage analysis, in turn, provides valuable feedback on the adequacy of testing efforts and the software's compliance with the specified requirements, making it an integral part of the development process in the automotive industry.
Impact of tools on code coverage
Compilers and software development tools can have an impact on code coverage during testing. The way code is compiled and the tools used for testing can influence the accuracy and completeness of code coverage results. Here are some ways in which compilers and tools can impact code coverage:
Optimization Level: Compilers often offer different optimization levels, such as -O0 (no optimization) to -O3 (high optimization). Higher optimization levels can lead to code transformations that may change the structure of the generated machine code, making it challenging to map the coverage results back to the original source code accurately. This can affect the reported code coverage metrics.
Debug Symbols: Debug symbols generated by the compiler are essential for mapping machine code back to the source code during debugging and code coverage analysis. Compiling without debug symbols can result in inaccurate or limited code coverage information.
Code Instrumentation: Code coverage tools often use instrumentation to track which parts of the code have been executed. Depending on the tool's implementation and the compiler used, the instrumentation process can introduce additional code, alter the original code layout, or affect program behavior, potentially influencing code coverage results.
Library and Framework Support: Some code coverage tools may have limited support for certain libraries or frameworks used in the software. This can result in reduced coverage for code executed within those libraries or frameworks.
Language Support: Code coverage tools may have varying levels of support for different programming languages. The capabilities and accuracy of coverage analysis can be influenced by the language being used.
Dynamic vs. Static Analysis: Code coverage tools can employ dynamic analysis (runtime instrumentation) or static analysis (source code analysis). The choice of method can affect the accuracy and types of code coverage metrics obtained.
Code Coverage Tool Configuration: The configuration of the code coverage tool, including the selection of coverage metrics (e.g., statement coverage, branch coverage), can impact the scope and accuracy of the coverage analysis.
To obtain accurate and reliable code coverage results, it is essential to use a well-configured code coverage tool, compile the code with appropriate compiler settings (debug symbols enabled), and consider any potential impacts of code instrumentation on the software's behavior. The selection of tools and compilers should align with the specific requirements and standards of the automotive industry, especially for safety-critical systems where code coverage is a critical aspect of software verification and validation.
Tools for code coverage
In the automotive industry, several code coverage tools are commonly used to measure the extent of code coverage during software testing and validation. These tools help assess how much of the source code has been executed by the test cases, ensuring thorough testing and verification of safety-critical systems. Some of the popular code coverage tools used in the automotive industry include:
VectorCAST/C++: VectorCAST/C++ is a widely used code coverage tool specifically designed for C and C++ code. It provides various coverage metrics such as statement coverage, branch coverage, function coverage, and more. VectorCAST/C++ is compliant with safety standards like ISO 26262 and IEC 61508.
Tessy (Razorcat): Tessy is a powerful unit and integration testing tool that includes code coverage analysis for automotive software written in C, C++, and Ada. It supports statement coverage, branch coverage, and MC/DC (Modified Condition/Decision Coverage) among other coverage metrics.
Klocwork (Perforce): Klocwork is a static code analysis tool that helps identify defects and potential vulnerabilities in the source code. While it is not a dedicated code coverage tool, it can be used alongside other tools to improve code quality and indirectly influence code coverage efforts.
LDRA Testbed: LDRA Testbed provides comprehensive code coverage analysis for software written in various languages, including C, C++, Ada, and more. It supports various coverage metrics and is often used in safety-critical industries such as automotive and aerospace.
PolySpace (MathWorks): PolySpace is a static analysis tool designed for embedded software development. While not primarily a code coverage tool, it helps identify code execution issues and defects that can impact code coverage results.
CoverageMaster (Verifysoft): CoverageMaster is a code coverage tool used to measure code coverage in C and C++ codebases. It supports statement, branch, and function coverage analysis.
Testwell CTC++ (Verifysoft): Testwell CTC++ is a code coverage tool that provides coverage analysis for C, C++, and C# code. It offers a wide range of coverage metrics, including branch and condition coverage.
RTRT (IBM Rational Test RealTime): RTRT is a real-time testing tool that also includes code coverage analysis capabilities. It supports various coverage metrics and is used in safety-critical domains such as automotive and aerospace.
These code coverage tools assist automotive software developers in assessing the quality and reliability of their software, ensuring compliance with safety standards, and achieving the required code coverage metrics to meet safety integrity levels (ASIL) specified in standards like ISO 26262. Organizations in the automotive industry typically select tools that align with their specific needs, development environment, and compliance requirements.
Impact of not doing optimal code coverage analysis
Not conducting code coverage analysis in the automotive industry can have significant implications for software safety, reliability, and compliance. Code coverage analysis is a critical aspect of software testing and verification, especially in safety-critical systems used in automobiles. Without adequate code coverage analysis, potential defects and safety issues may go undetected, leading to serious consequences. Let's explore an actual example to illustrate the impact of not performing code coverage analysis in the automotive industry:
Example Scenario:
Suppose an automotive manufacturer is developing the software for an advanced driver-assistance system (ADAS) that includes features like adaptive cruise control and lane-keeping assistance. The software is intended to control the vehicle's speed and steering in certain driving conditions to enhance safety and driver comfort.
Impact of Not Doing Code Coverage Analysis:
Undetected Bugs and Safety Issues:
Without code coverage analysis, certain parts of the software may remain untested or under-tested. As a result, potential bugs, logic errors, or safety-critical issues may go unnoticed in these untested areas.
For instance, if code coverage analysis is not performed on the lane-keeping assistance feature, a critical bug that causes the vehicle to drift out of the lane without appropriate corrective action might not be detected during testing.
Risk of Safety Incidents on the Road:
Inadequate testing and code coverage increase the risk of software failures in real-world driving scenarios.
For instance, if the adaptive cruise control feature is not thoroughly tested for edge cases (e.g., sudden obstacle appearance), the system may fail to respond appropriately, leading to potential collisions or safety incidents.
Non-compliance with Safety Standards:
The automotive industry is regulated by safety standards such as ISO 26262, which require sufficient code coverage analysis to ensure the safety of the software.
A failure to perform code coverage analysis and meet the prescribed code coverage targets can result in non-compliance with these safety standards, leading to legal and reputational consequences for the manufacturer.
Increased Maintenance Costs:
Undetected defects and software issues may surface after the software is deployed in vehicles.
Addressing these issues in the field can be time-consuming and expensive, leading to increased maintenance costs for the automotive manufacturer.
Loss of Customer Trust:
Software failures and safety issues can lead to recalls and negative media coverage, eroding customer trust in the manufacturer's products.
Customers may lose confidence in the brand, impacting future sales and market reputation.
In summary, not conducting code coverage analysis in the automotive industry can have severe repercussions on software safety, quality, and compliance. It is essential for automotive manufacturers to employ rigorous testing practices, including code coverage analysis, to identify and address potential issues before the software is deployed in vehicles. Code coverage analysis is an integral part of ensuring the reliability and safety of software used in modern automobiles.