10. Coverage Stats

This section will discuss how to generate a coverage statistics report of a given suite using RISCV_ISAC.

If you have followed the Quickstart steps of this guide then riscv_isac should already be installed on your system. If not then please follow the installation guide provided here to install riscv_isac

Coverage statistics are obtained by executing the tests on the SAIL-RISCV model and parsing the execution logs of each test generated by the model. Coverage is only covered for those tests which are selected via the configuration file provided.

10.1. Usage

In the config.ini script set the reference model as sail_cSim. The ISA YAML of a DUT plugin is used to check for coverage. A sample config.ini is shown below:

[RISCOF]
ReferencePlugin=cSail
ReferencePluginPath=/home/neel/temp/riscof-plugins/sail_cSim
DUTPlugin=spike
DUTPluginPath=/home/neel/temp/riscof-plugins/spike

[spike]
pluginpath=/home/neel/temp/riscof-plugins/spike
ispec=/home/neel/temp/riscof-plugins/spike/spike_isa.yaml
pspec=/home/neel/temp/riscof-plugins/spike/spike_platform.yaml

[cSail]
pluginpath=/home/neel/temp/riscof-plugins/sail_cSim

To run coverage

$ riscof --verbose debug coverage --suite /path/to/suite --env /path/to/suite

The log of the above command is shown below:

INFO | [--root--]: Reading configuration from: /home/neel/temp/config.ini
INFO | [--root--]: Preparing Models
INFO | [--riscv_config.checker--]: Input-ISA file
INFO | [--riscv_config.checker--]: Loading input file: /home/neel/temp/riscof-plugins/spike/spike_isa.yaml
INFO | [--riscv_config.checker--]: Load Schema /home/neel/.pyenv/versions/3.7.0/envs/venv/lib/python3.7/site-packages/riscv_config/schemas/schema_isa.yaml
INFO | [--riscv_config.checker--]: Processing Hart: hart0
INFO | [--riscv_config.checker--]: Initiating Validation
INFO | [--riscv_config.checker--]: No errors for Hart: 0 :)
INFO | [--riscv_config.checker--]: Initiating post processing and reset value checks.
INFO | [--riscv_config.checker--]: Dumping out Normalized Checked YAML: /home/neel/temp/riscof_work/spike_isa_checked.yaml
INFO | [--riscv_config.checker--]: Input-Platform file
INFO | [--riscv_config.checker--]: Loading input file: /home/neel/temp/riscof-plugins/spike/spike_platform.yaml
INFO | [--riscv_config.checker--]: Load Schema /home/neel/.pyenv/versions/3.7.0/envs/venv/lib/python3.7/site-packages/riscv_config/schemas/schema_platform.yaml
INFO | [--riscv_config.checker--]: Initiating Validation
INFO | [--riscv_config.checker--]: No Syntax errors in Input Platform Yaml. :)
INFO | [--riscv_config.checker--]: Dumping out Normalized Checked YAML: /home/neel/temp/riscof_work/spike_platform_checked.yaml
INFO | [--root--]: Will collect Coverage using RISCV-ISAC
INFO | [--root--]: CGF file being used : /scratch/git-repo/incoresemi/riscof/riscof/suite/compliance.cgf
INFO | [--riscof.framework.test--]: Selecting Tests.
INFO | [--riscof.framework.main--]: Running Tests on Reference.
INFO | [--riscof.framework.main--]: Merging Coverage reports
INFO | [--root--]: Test report generated at /home/neel/temp/riscof_work/coverage.html.
INFO | [--root--]: Openning test report in web-browser

The above command will open an html report in your default browser with coverage statistics.

10.2. Coverage for new tests

If you would like to test coverage of new added tests then you can either provide a new CGF file via the command line --cgf <path to cgf file> or update the cgf file present in riscof/suite/compliance.cgf if you have installed RISCOF in dev mode.