9. Test Format Spec

9.1. Vocabulary

9.1.1. The architectural test

The architectural test is a nonfunctional testing technique which is done to validate whether the system developed meets the prescribed standard or not. In this particular case the golden reference is the RISC-V ISA standard.

For purpose of this document we understand that the architectural test is a single test which represents the minimum test code that can be compiled and run. It is written in assembler code and its product is a test signature. An architectural test may consist of several test cases.

9.1.2. The RISC-V architectural test pool

The RISC-V architectural test pool consists of all approved architectural tests that can be assembled by the test framework, forming the architectural test suite. The RISC-V architectural test pool must be test target independent (so, should correctly run on any compliant target). Note that this nonfunctional testing is not a substitute for verification or device test.

9.1.3. The RISC-V architectural test suite

The RISC-V architectural test suite is a group of tests selected from the architectural test pool to test architectural for the specific RISC-V configuration. Test results are obtained in the form of a test suite signature. Selection of tests is performed based on the target’s asserted configuration, and the spec, Execution Environment or platform requirements. Compliant processor or processor models shall exhibit the same test suite signature as the golden reference test suite signature for the specific configuration being tested.

9.1.4. The test case

A test case is part of the architectural test that tests just one feature of the specification.

Note

a single test can contain multiple test cases, each of which can have its own test inclusion condition (as defined by the cond_str parameter of the RVTEST_CASE macro.

testStruct

9.1.5. The test case signature

The test case signature is represented by single or multiple values. Values are written to memory at the address starting at the address specified by the RVMODEL_DATA_BEGIN and ending at RVMODEL_DATA_END. Signatures can be generated most easily using the RVTEST_SIGUPD macro.

9.1.6. The test signature

The test signature is a characteristic value which is generated by the architectural test run. The test signature may consist of several test case signature, prefixed with a separate line containing the name of the test and a unique value indicating its version (e.g. git checkin hash). The test target is responsible for extracting values from memory and properly formatting them, using metadata provided to it by the framework using the RVMODEL_DATA_BEGIN and RVMODEL_DATA_END macros. Test case signature values are written one per line, starting with the most-significant byte on the left-hand side with the format <hex_value> where the length of value will be 32 bits (so 8 characters), regardless of the actual value length computed by the test.

9.1.7. The test suite signature

The test suite signature is defined as a set of test signature valid for given computed test suite. It represents the test signature of the particular RISC-V configuration selected for the architectural test.

9.1.8. The target shell

The target shell is a vendor-supplied software and hardware environment around the test target that enables it to communicate with the framework, including assembling and linking tests, loading tests into memory, executing tests, and extracting the signature. The input to the target shell is a .S architectural test file, and the output is a test signature.

9.1.9. The test target

The test target can include a RISC-V Instruction Set Simulator (ISS), a RISC-V emulator, a RISC-V RTL model running on an HDL simulator, a RISC-V FPGA implementation or a physical chip or ohter model type. Each of the target types offers specific features and represents specific interface challenges. It is a role of the target shell to handle different targets while using the same architectural test pool as a test source.

9.1.10. The architectural test framework

The architectural test framework selects and configures the architectural test suite from the architectural test pool for the selected test target based on both the specific architectural choices made by an implementation and those required by the Execution Environment. It causes the target shell to build, execute, and report a signature. The architectural test framework then compares reported signatures, inserts test part names and version numbers and summarizes differences (or lack of them) into a RISC-V architectural report. The primary role of the well-defined architectural test pool structure is to provide the tests in a form suitable for the Compliance Test Framework selection engine.

9.2. Compliance test pool

9.2.1. Test pool structure

The structure of architectural test in the architectural test pool shall be based on defined RISC-V extensions and privileged mode selection. This will provide a good overview of which parts of the ISA specification are already covered in the architectural test suite, and which tests are suitable for certain configurations. The architectural test pool has this structure:

architectural-tests-suite (root)
-- <architecture>_<mode>/<feature(s)>, where

<architecture> is [ RV32I | RV64I | RV32E ]

<mode> is [ M | MU | MS | MSU ], where
   M   Machine      mode tests - tests execute in M-mode only
   MU  Machine/User mode tests - tests execute in both M- & U-modes (S-mode may exist)
   MS  Machine/Supv mode tests - tests execute in both M- & S-modes (not U-mode)
   MSU All          mode tests - tests execute in all of M-, S-, & U-Modes

<feature(s)> are the lettered extension [A | B | C | M ...] or subextension [Zifencei | Zam | ...]
when the tests involve extensions, or more general names when tests cut across extension
definitionss (e.g. Priv, Interrupt, Vm). The feature string consists of an initial capital
letter, followed by any further letters in lower case.

Note

this structure is for organizational purposes, not functional purposes, although full test names will take advantage of it.

Tests that will be executed in different modes, even if the results are identical, should be replicated in each mode directory, e.g. RV32I_M/, RV32I_MS/, and RV32I_MU/. These tests are typically those involving trapping behavior, e.g load, store, and privilged ops.

9.2.2. Test naming

The naming convention of a single test:

<test objective>-<test number>.S

  • test objective - an aspect that the test is focused on. A test objective may be an instruction for ISA tests (ADD, SUB, …), or a characteristic covering multiple instructions, e.g. exception event (misaligned fetch, misalign load/store) and others.

  • test number - number of the test. It is expected that multiple tests may be specified for one test objective. We recommend to break down complex tests into a set of small tests. A simple rule of thumb is one simple test objective = one simple test. The code becomes more readable and the test of the objective can be improved just by adding test case. The typical example are instruction tests for the F extension.

  • A test name shall not include an ISA category as part of its name (i.e. the directory, subdirectory names). Experience has shown that including ISA category in the test name leads to very long test names. Instead, we have introduced the Test pool structure where the full name is composed of the test path in the Test pool structure and the simple test name. Since full names can be reconstructed easily it is not necessary to include the path in test names.

9.2.3. The test structure of an architectural test

All tests shall use a signature approach. Each test shall be written in the same style, with defined mandatory items.The test structure of an architectural test shall have the following sections in the order as follows:

  • Header + license (including a specification link, a brief test description and RVTEST_ISA macro))

  • Includes of header files (see Common Header Files section)

  • Test Virtual Machine (TVM) specification

  • Test code between “RVTEST_CODE_BEGIN” and “RVTEST_CODE_END”

  • Input data section, marked with “RVMODEL_DATA_SECTION”

  • Output data section between “RVMODEL_DATA_BEGIN” and “RVMODEL_DATA_END”.

Note

there is no a requirement that the code or scratch data sections must be contiguous in memory, or that they be located before or after data or code sections (configured by embedded directives recognized by the linker)

9.2.4. Available Macros

There are both pre-defined and model-specific macros which shall be used in every test to guarantee their portability. In addition, there are both pre-defined and model specific macros that are not required, but may be used in tests.

  • Required, Pre-defined Macros

    • RVTEST_ISA(isa_str) :

      • defines the Test Virtual Machine (TVM, the ISA being tested)

      • Empty macro to specify the isa required for compilation of the test.

      • This is mandated to be present at the start of the test.

    • RVTEST_CODE_BEGIN :

      • start of code (test) section

      • Macro to indicate test code start add and where test startup routine is inserted.

      • No part of the code section should precede this macro

    • RVTEST_CODE_END :

      • end of code (test) section

      • Macro to indicate test code end.

      • No part of the code section should follow after this macro.

    • RVTEST_CASE(CaseName, CondStr) :

      • execute this case only if condition in cond_str are met

      • CaseName is arbitrary string

      • CondStr is evaluated to determine if the test-case is enabled and sets name variable

      • CondStr can also define compile time macros required for the test-case to be enabled.

      • the test-case must be delimited with an #ifdef CaseName/#endif pair

      • the format of CondStr can be found in https://riscof.readthedocs.io/en/latest/cond_spec.html#cond-spec

  • Required, Model-defined Macros

    • RVMODEL_DATA_BEGIN:

      • start of output data (signature) section

    • RVMODEL_DATA_END:

      • end of output data (signature) section

    • RVMODEL_DATA_SECTION:

      • model defined data area

      • contains static input data and intermediate scratch area for the test (e.g. stack)

    • RVMODEL_HALT:

      • defines model halt mechanism, which starts signature saving

  • Optional, Pre-defined Macros

    • RVTEST_SIGBASE(BaseReg,Val):

      • defines the base register used to update signature values

      • Register BaseReg is loaded with value Val

      • hidden_offset is initialized to zero

    • RVTEST_SIGUPD(BaseReg, SigReg [,Value, TmpReg]):

      • Register SigReg is stored in mem(BaseReg+hidden_offset)

      • hidden_offset is post incremented so repeated uses store signature values sequentially

      • Optionally, if Value and TmpReg are specified and assertions are enabled, RVMODEL_IO_ASSERT_GPR_EQ(SigReg, TmpReg, Value) is enabled.

    • RVTEST_BASEUPD(BaseReg[oldBase[,newOff]]):

      • [moves &] updates BaseReg past stored signature

      • Register BaseReg is loaded with the oldReg+newOff+hidden_offset

      • BaseReg is used if oldBase isn’t specified; 0 is used if newOff isn’t specified

      • hidden_offset is re-initialized to 0 afterwards

  • Optional, Model-defined Macros

    • RVMODEL_BOOT:

      • contains boot code for model; may include emulation code or trap stub

    • RVMODEL_IO_INIT:

      • initializes IO for debug output

      • this must be invoked if any of the other RV_MODEL_IO_* macros are used

    • RVMODEL_IO_CHECK:

      • checks IO for debug output

      • <needs description of how this is used >

    • RVMODEL_IO_ASSERT_GPR_EQ(ScrReg, Reg, Value):

      • debug assertion that GPR should have value

      • outputs a debug message if Reg!=Value

      • ScrReg is a scratch register used by the output routine; its final value cannot be guaranteed

    • RVMODEL_IO_WRITE_STR(ScrReg, String):

      • output debug string, using a scratch register

      • outputs the message String

      • ScrReg is a scratch register used by the output routine; its final value cannot be guaranteed

Note

Note that there is no a requirement that the code or scratch data sections must be contiguous in memory, or that they be located before or after data or code sections (configured by embedded directives recognized by the linker)

9.2.5. Common test format rules

There are the following common rules that shall be applied to each architectural test: 1. Always use // as commentary. # should be used only for includes and defines. 2. A test shall be divided into logical blocks (test case) according to the test goals. Test cases are enclosed in an #ifdef <__CaseName__>, #endif pair and begin with the RVTEST_CASE(CaseName,CondStr) macro that specifies the test case name, and a string that defines the conditions under which that test case can be selected for assembly and execution. Those conditions will be collected and used to generate the database which in turn is used to select tests for inclusion in the test suite for this target. 3. Tests should use the RVTEST_SIGBASE(BaseReg,Val) macro to define the GPR used as a pointer to the output signature area, and its initial value. It can be used multiple times within a test to reassign the output area or change the base register. This value will be used by the invocations of the RVTEST_SIGUPD macro. 4. Tests should use the RVTEST_SIGUPD(BaseReg, SigReg, ScratchReg, Value) macro to store signature values using (only) the base register defined in the most recently encountered RVTEST_SIGBASE(BaseReg,Val) macro. Repeated uses will automatically have an increasing offset that is managed by the macro.

  • Uses of RVTEST_SIGUPD shall always be preceded sometime in the test case by RVTEST_SIGBASE.

  • The RVTEST_SIGUPD macro may optionally invoke a test assertion macro (e.g. RVMODEL_IO_ASSERT_GPR_EQ) with an assertion value for debugging, determined by the presence of ScratchReg and Value parameters.

  • Tests that use SIGUPD inside a loop or in any section of code that will be repeated (e.g. traps) must use the BASEUPD macro between each loop iteration or repeated code to ensure static values of the base and offset don’t overwrite older values.

  1. When macros are needed for debug purposes, only macros from compliance_model.h shall be used. Note that using this feature shall not affect the signature results of the test run.

  2. Test shall not include other tests (e.g. #include “../add.S”) to prevent non-complete tests, compilation issues, and problems with code maintenance.

  3. Tests and test cases shall be skipped if not required for a specific model test configuration based on test conditions defined in the RVTEST_CASE macro. Tests that are selected may be further configured using variables (e.g. XLEN) which are passed into the tests and used to compile them. In either case, those conditions and variables are derived from the YAML specification of the device and execution environment that are passed into the framework. The flow is to run an architectural test suite built by the The architectural test framework from the The RISC-V architectural test pool to determine which tests and test cases to run.

  4. Tests shall not depend on tool specific features. For example, tests shall avoid usage of internal GCC macros (e..g. ____risc_xlen__), specific syntax (char ‘a’ instead of ‘a) or simulator features (e.g. tohost) etc.

  5. A test will end by either jumping to or implicitly reaching the RVTEST_CODE_END macro (i.e. rvtest_code_end label). The RVTEST_CODE_END macro is always followed by the RVMODEL_HALT macro.

  6. Macros defined outside of a test shall only be defined in specific predefined header files (see Common Header Files below), and once they are in use, they may be modified only if the function of all affected tests remains unchanged. It is acceptable that macros use may lead to operand repetition (register X is used every time).

  • The aim of this restriction is to have test code more readable and to avoid side effects which may occur when different contributors will include new The architectural test or updates of existing ones in the The RISC-V architectural test pool. This measure results from the negative experience, where the The RISC-V architectural test suite could be used just for one target while the architectural test code changes were necessary to have it also running for other targets.

  1. All contents of the signature region must always be initialized to 0xdeadbeef.

  2. The result of no operation (that is going to be stored in the signature) should be 0xdeadbeef.

  3. Pseudo ops other than li and la which can map to multiple standard instruction sequences should not be used.

  4. The actual test-section of the assembly must always start with the RVTEST_CODE_BEGIN which contains a routine to initialize the registers to specific values.

9.2.6. Common Header Files

Each test shall include only the following header files:

  • compliance_model.h – defines target-specific macros, both required and optional: (e.g. RVMODEL_xxx)

  • compliance_test.h – defines pre-defined test macros both required and optional: (e.g. RVTEST_xxx)

Adding new header files is forbidden. It may lead to macro redefinition and compilation issues. Macros maybe defined and used inside a test, as they will not be defined outside that specific test. Assertions will generate code that reports assertion failures (and optionally successes?) only if enabled by the framework. In addition, the framework may collect the assertion values and save them as a signature output file if enabled by the framework.

9.2.7. RVTEST_CASE Condition Formating

This section describes the format for the conditions CondStr to be followed while writing the RVTEST_CASE macro. Each of the statements within this macro ends with a ‘;’ .

The macro follows the convention mentioned here. This section describes the syntax to be followed by the CondStr of the RVTEST_CASE macro.

Note

A keylist is a string of ‘>’ separated words(keys) which is used to navigate the supplied specs. The schema may be used to specify them. Only valid keys and their combinations are allowed(as present in the scema).

There are two types of valid statements allowed.

  1. check statements

    These statements get translated into the condtions which need to be true for the part to be enabled. The condition can be structured in one of the following allowed ways.

    • keylist:=value

      The keylist specifies the path to the field in the ISA YAML dictionary whose value needs to be checked. The value is the value against which the entry in the input yaml is checked. The value can be a regular expression as well, in which case it should be specified as regex(“expression”)

      Example:

      check ISA:=regex(.*I.*Zicsr.*);  # checks if ISA node supports I and Zicsr extensions.

check hw_data_misaligned_support:=True; # checks if the misaligned support is available.

    • keylist=key

      The keylist specifies the path to the field whose keys needs to be checked. The key is the key whose presence needs to be checked in the field specified by the keylist.

      Example:

      check mtvec>rv32>base>type=warl; # checks if mtvec is a warl field

    • function_call=Rval

      The function_call specifies the function to be called along with the arguments to be specified to the function. The node from the yaml which has to be passed to the function can be specified using the keylist. Rval is the value against which the return value of the function is checked. The list of different functions,arguments and their return values is listed below.

      Function Signatures

      • writable(bit_position,keylist_for_field) -> bool

        Checks whether the bit at bit_position for a particular csr_field_name is writable. This function is typically used for WARL nodes.

      • islegal(value,dependency_values_list,keylist_for_field) -> bool

        This function is valid only for WARL fields in the csrs. Checks whether the value is a legal value when the values of the fields listed as dependency for the field in question on is equal to the dependency_values_list.

      Example:

      check writable(12,misa>rv32>extensions)=True; # checks if 12th bit in MISA is writable.

  2. def statements

    def macro(s)(=value/keylist/function);

    These statements specify which macros to be defined for the part to run and their values(optional). * The macro specifies the name of the macro. * Multiple macros can be specified using a comma inbetween them. * A keylist specifying the path of the field whose value has to be passed as the value of the macro can also be given. * A function along with the arguments can also be specified. At runtime the function is called using the specified arguments and its return values are assigned to the macro(s) specified. The list of functions supported are as follows.

    Function Signatures

    • getlegal(dependency_values_list,num_vals,key_list_for_field) -> list(int)

      This function is valid only for WARL fields in the csrs. It returns a list of legal values for the specified field when the values of the fields listed as dependency for the field in question on is equal to the dependency_values_list. The length of the list returned is equal to num_vals. Each entry in the list is assigned to the corresponding macro listed on the left hand side of the = sign.

    • getillegal(dependency_values_list,num_vals,key_list_for_field) -> list(int)

      This function is valid only for WARL fields in the csrs. It returns a list of illegal values for the specified field when the values of the fields listed as dependency for the field in question on is equal to the dependency_values_list. The length of the list returned is equal to num_vals. Each entry in the list is assigned to the corresponding macro listed on the left hand side of the = sign.

    Example:

    def TEST_CASE_1=True; # enables TEST_CASE_1 macro during compilation phase.

def rvtest_mtrap_routine=True; #enabled trap routines during compilation phase.

# Assigns a legal value(for the base field in mtvec) to LEGAL_2_1
def LEGAL_2_1 = getlegal([0],1,mtvec>rv32>base);

# Assigns an illegal value(for the base field in mtvec) each to ILLEGAL_2_1 and ILLEGAL_2_2
def ILLEGAL_2_1,ILLEGAL_2_2 = getillegal([0],2,mtvec>rv32>base);