Module InlineExpectationsTest

Provides a library for writing QL tests whose success or failure is based on expected results embedded in the test source code as comments, rather than the contents of an .expected file (in that the .expected file should always be empty).

To add this framework to a new language, add a new file (usually called InlineExpectationsTest.qll) with:

• private import codeql.util.test.InlineExpectationsTest (this file)
• An implementation of the signature in InlineExpectationsTestSig. Usually this is done in a module called Impl. Impl has to define a Location class, and an ExpectationComment class. The ExpectationComment class must support a getContents method that returns the contents of the given comment, excluding the comment indicator itself. It should also define toString and getLocation as usual.
• import Make<Impl> to expose the query predicates constructed in the Make module.

To create a new inline expectations test:

• Declare a module that implements TestSig, say TestImpl.
• Implement the hasActualResult() predicate to produce the actual results of the query. For each result, specify a Location, a text description of the element for which the result was reported, a short string to serve as the tag to identify expected results for this test, and the expected value of the result.
• Implement getARelevantTag() to return the set of tags that can be produced by hasActualResult(). Often this is just a single tag.
• import MakeTest<TestImpl> to ensure the test is evaluated.

Example:

module ConstantValueTest implements TestSig {
  string getARelevantTag() {
    // We only use one tag for this test.
    result = "const"
  }

  predicate hasActualResult(
    Location location, string element, string tag, string value
  ) {
    exists(Expr e |
      tag = "const" and // The tag for this test.
      value = e.getValue() and // The expected value. Will only hold for constant expressions.
      location = e.getLocation() and // The location of the result to be reported.
      element = e.toString() // The display text for the result.
    )
  }
}

import MakeTest<ConstantValueTest>

There is no need to write a select clause or query predicate. All of the differences between expected results and actual results will be reported in the testFailures() query predicate.

To annotate the test source code with an expected result, place a comment starting with a $ on the same line as the expected result, with text of the following format as the body of the comment:

tag=expected-value

Where tag is the value of the tag parameter from hasActualResult(), and expected-value is the value of the value parameter from hasActualResult(). The =expected-value portion may be omitted, in which case expected-value is treated as the empty string. Multiple expectations may be placed in the same comment. Any actual result that appears on a line that does not contain a matching expected result comment will be reported with a message of the form “Unexpected result: tag=value”. Any expected result comment for which there is no matching actual result will be reported with a message of the form “Missing result: tag=expected-value”.

Example:

int i = x + 5;  // $ const=5
int j = y + (7 - 3)  // $ const=7 const=3 const=4  // The result of the subtraction is a constant.

For tests that contain known missing and spurious results, it is possible to further annotate that a particular expected result is known to be spurious, or that a particular missing result is known to be missing:

$ SPURIOUS: tag=expected-value // Spurious result $ MISSING: tag=expected-value // Missing result

A spurious expectation is treated as any other expected result, except that if there is no matching actual result, the message will be of the form “Fixed spurious result: tag=value”. A missing expectation is treated as if there were no expected result, except that if a matching expected result is found, the message will be of the form “Fixed missing result: tag=value”.

A single line can contain all the expected, spurious and missing results of that line. For instance: $ tag1=value1 SPURIOUS: tag2=value2 MISSING: tag3=value3.

If the same result value is expected for two or more tags on the same line, there is a shorthand notation available:

tag1,tag2=expected-value

is equivalent to:

tag1=expected-value tag2=expected-value