4. RISCOF Commands

This section provides an overview and working of the various sub commands available in RISCOF. The current list of subcommands includes:

  • arch-tests

  • coverage

  • gendb

  • setup

  • validateyaml

  • testlist

  • run

4.1. arch-tests

This command is used to clone and update the tests from the official riscv-arch-test repository.

This command requires one of the following flags to be specified from the cli.

  • show-version: Display the current version of the official suite present at the specified directory path.

  • clone: Clone the suite from github.

  • update: Update the suite to reflect latest changes from github.

Optional arguments from the cli:

  • get-version: The specific version of the tests to be fetched. Can be used with both the clone and update flags. The latest release is fetched if skipped.

  • dir: The path to the directory where the suite is to be cloned to. Defaults to ./riscv-arch-test if skipped.

4.2. coverage

This command is used to collect the ISA coverage metrics of a given test-suite and generate a coverage report in html.

This command will require the following inputs from the cli:

  • suite: The test suite path on which coverage needs to be run

  • env: The path to the environment directory containing the suite-specific header files.

  • cgf: list of covergroup-format files specifying the coverpoints that need to be covered by the the suite

Optional arguments from the cli:

  • config: path to the config.ini file. Defaults to ./config.ini if skipped.

  • work-dir: path to the working directory where all artifacts need to be dumped. Defaults to ./riscof_work

  • no-browser: when used, RISCOF skips automatically opening the html report in the default web browser.

The coverage command simply passes the cgf files to the reference plugin’s runTests function. The Reference plugin is responsible to generating a yaml based coverage report for each test using riscv-isac. The yaml file should be named coverage.rpt. The riscv-isac run will also generate a data-propagation report which should be named as ref.md.

Once the coverage files for each test has been generated, RISCOF will parse through the working directories and merge all the coverage.rpt files to create a single yaml coverage report: suite_coverage.rpt. RISCOF then also converts this to an HTML based reports and open it on the default web-browser.

For a example on using this feature please refer to the Coverage Stats section.

4.3. gendb

This command is used to generate a database yaml file for all tests available in the test-suite. The commands requires the following inputs from the cli:

  • suite: The test suite path for which database needs to be generated.

  • work-dir: path to the working directory where all artifacts need to be dumped. Defaults to ./riscof_work

This utility parses the suite directory and collects all the .S files. For each .S file, the utility will parse the test and collect informations from various macros such as RVTEST_ISA, RVTEST_CASE, etc. For each test the utility will create a new entry in a dictionary which captures the different parts of the tests, the enabling conditions of each part, the coverage contributions of each part, any compile macros required for each part and muc more.

The generated database yaml will follow the syntax described in section Database Generator.

The output of this utility is a database.yaml located in the work_dir directory. This file is used by RISCOF to select and filter tests based on input DUT configuration.

Note

The tests that are parsed by the gendb utility must follow the TestFormat Spec set forth by the riscv-arch-test SIG.

4.4. setup

The setup command is used to generate a series of Template files that are required by RISCOF. These files are meant to provide ease to users integrating their DUT to RISCOF for the first time and should be modified by the users.

The setup utility takes in the following optional inputs from the cli:

  • dutname: name of the dut for running the tests on. The utility will use this name to create a template plugin directory with all the relevant files. These files will have to be modified by the user. Defaults to “spike” when skipped.

  • refname: name of the reference plugin to be used in RISCOF. The utility will use this name to create a reference plugin directory with all the relevant files.

The setup utility will also create a sample config.ini file using the above inputs.

4.5. validateyaml

This command simply performs a validation of the isa spec and the platform pspec yamls of the DUT as mentioned in the config.ini using riscv-config. The outputs are checked version of the yamls in the directory pointed by work_dir

4.6. testlist

This command is used to filter tests from the database.yaml based on the configuration of DUT present in the isa and platform yamls as mentioned in the config.ini. This command will require the following inputs from the cli:

  • suite: The test suite from which the test need to filtered.

This command takes the following optional inputs from cli

  • config: path to the config.ini file. Defaults to ./config.ini if skipped.

  • work-dir: path to the working directory where all artifacts need to be dumped. Defaults to ./riscof_work

The utility first creates a database.yaml for the input suite. For each test in the database yaml, this utility will check if the conditions of any parts of a test are enabled based on the isa and platform yaml specs of the DUT. If any part is enabled, then the corresponding test is entered into the teslist along with the respective coverage labels and compile macros.

The utility will dump the test list in the testlist.yaml file in the work_dir directory. This yaml will follow the same syntax as defined in the Test List Format section.

4.7. run

This is probably the primary command of RISCOF which is going to be widely used. This command is currently responsible for first validating the inputs yamls, creating a database of the tests in the suite directory, generate a filtered test-list, run the tests on the DUT and then the Reference Plugins, and finally compare the generated signatures and present an html report of the findings.

The following inputs are required on the cli by this command:

  • suite: The test suite path on which coverage needs to be run

  • env: The path to the environment directory containing the suite-specific header files.

Optional arguments from the cli:

  • config: path to the config.ini file. Defaults to ./config.ini if skipped.

  • work-dir: path to the working directory where all artifacts need to be dumped. Defaults to ./riscof_work

  • no-browser: when used, RISCOF skips automatically opening the html report in the default web browser.

  • dbfile: The path to the database file, from which testlist will be generated

  • testfile: The path to the testlist file on which tests will be run

  • no-ref-run: when used, RISCOF will not run tests on Reference and will quit before signatures comparison

  • no-dut-run: when used, RISCOF will not run tests on DUT and will quit before signatures comparison

  • no-clean: when used, RISCOF will not remove the work_dir, if it exists.

The work_dir is cleaned by default. However, if one of no-clean, testfile or dbfile are specified, it is preserved as is.

All artifacts of this command are generated in the work_dir directory. Typicall artifacts will include:

Artifact

Description

database.yaml

This is the database of all the tests in the suite directory

Makefile.DUT*

This is the Makefile generated by the DUT Plugin.

Makefile.Reference*

This is the Makefile generated by the Reference Plugin.

report.html

The final report generated at the end of the run after signature comparison

yaml files

verified and checked yaml versions of the input isa and platform yamls

test_list.yaml

This list of filtered tests from the database.yaml

src directory

this will include a directory for each test in the test_list.yaml. Each test-directory will include the test, compiled-binaries, signatures from both the DUT and the Reference Plugin.