Expressions and statements in Python¶

You can use syntactic classes from the CodeQL library to explore how Python expressions and statements are used in a code base.

Statements¶

The bulk of Python code takes the form of statements. Each different type of statement in Python is represented by a separate CodeQL class.

Here is the full class hierarchy:

Stmt – A statement
- Assert – An assert statement
- Assign
  - AssignStmt – An assignment statement, x = y
  - ClassDef – A class definition statement
  - FunctionDef – A function definition statement
- AugAssign – An augmented assignment, x += y
- Break – A break statement
- Continue – A continue statement
- Delete – A del statement
- ExceptStmt – The except part of a try statement
- Exec – An exec statement
- For – A for statement
- Global – A global statement
- If – An if statement
- ImportStar – A from xxx import * statement
- Import – Any other import statement
- Nonlocal – A nonlocal statement
- Pass – A pass statement
- Print – A print statement (Python 2 only)
- Raise – A raise statement
- Return – A return statement
- Try – A try statement
- While – A while statement
- With – A with statement

Example finding redundant ‘global’ statements¶

The global statement in Python declares a variable with a global (module-level) scope, when it would otherwise be local. Using the global statement outside a class or function is redundant as the variable is already global.

import python

from Global g
where g.getScope() instanceof Module
select g

The line: g.getScope() instanceof Module ensures that the Scope of Global g is a Module, rather than a class or function.

Example finding ‘if’ statements with redundant branches¶

An if statement where one branch is composed of just pass statements could be simplified by negating the condition and dropping the else clause.

if cond():
    pass
else:
    do_something

To find statements like this that could be simplified we can write a query.

import python

from If i, StmtList l
where (l = i.getBody() or l = i.getOrelse())
  and forall(Stmt p | p = l.getAnItem() | p instanceof Pass)
select i

Many codebases have some if statements that match this pattern.

The line: (l = i.getBody() or l = i.getOrelse()) restricts the StmtList l to branches of the if statement.

The line: forall(Stmt p | p = l.getAnItem() | p instanceof Pass) ensures that all statements in l are pass statements.

Expressions¶

Each kind of Python expression has its own class. Here is the full class hierarchy:

Expr – An expression
- Attribute – An attribute, obj.attr
- BinaryExpr – A binary operation, x+y
- BoolExpr – Short circuit logical operations, x and y, x or y
- Bytes – A bytes literal, b"x" or (in Python 2) "x"
- Call – A function call, f(arg)
- Compare – A comparison operation, 0 < x < 10
- Dict – A dictionary literal, {'a': 2}
- DictComp – A dictionary comprehension, {k: v for ...}
- Ellipsis – An ellipsis expression, ...
- GeneratorExp – A generator expression
- IfExp – A conditional expression, x if cond else y
- ImportExpr – An artificial expression representing the module imported
- ImportMember – An artificial expression representing importing a value from a module (part of an from xxx import * statement)
- Lambda – A lambda expression
- List – A list literal, ['a', 'b']
- ListComp – A list comprehension, [x for ...]
- Name – A reference to a variable, var
- Num – A numeric literal, 3 or 4.2
  - FloatLiteral
  - ImaginaryLiteral
  - IntegerLiteral
- Repr – A backticks expression, x (Python 2 only)
- Set – A set literal, {'a', 'b'}
- SetComp – A set comprehension, {x for ...}
- Slice – A slice; the 0:1 in the expression seq[0:1]
- Starred – A starred expression, *x in the context of a multiple assignment: y, *x = 1,2,3 (Python 3 only)
- StrConst – A string literal. In Python 2 either bytes or unicode. In Python 3 only unicode.
- Subscript – A subscript operation, seq[index]
- UnaryExpr – A unary operation, -x
- Unicode – A unicode literal, u"x" or (in Python 3) "x"
- Yield – A yield expression
- YieldFrom – A yield from expression (Python 3.3+)

Example finding comparisons to integer or string literals using ‘is’¶

Python implementations commonly cache small integers and single character strings, which means that comparisons such as the following often work correctly, but this is not guaranteed and we might want to check for them.

x is 10
x is "A"

We can check for these using a query.

import python

from Compare cmp, Expr literal
where (literal instanceof StrConst or literal instanceof Num)
  and cmp.getOp(0) instanceof Is and cmp.getComparator(0) = literal
select cmp

The clause cmp.getOp(0) instanceof Is and cmp.getComparator(0) = literal checks that the first comparison operator is “is” and that the first comparator is a literal.

Tip

We have to use cmp.getOp(0) and cmp.getComparator(0)as there is no cmp.getOp() or cmp.getComparator(). The reason for this is that a Compare expression can have multiple operators. For example, the expression 3 < x < 7 has two operators and two comparators. You use cmp.getComparator(0) to get the first comparator (in this example the x) and cmp.getComparator(1) to get the second comparator (in this example the 7).

Example finding duplicates in dictionary literals¶

If there are duplicate keys in a Python dictionary, then the second key will overwrite the first, which is almost certainly a mistake. We can find these duplicates with CodeQL, but the query is more complex than previous examples and will require us to write a predicate as a helper.

import python

predicate same_key(Expr k1, Expr k2) {
  k1.(Num).getN() = k2.(Num).getN()
  or
  k1.(StrConst).getText() = k2.(StrConst).getText()
}

from Dict d, Expr k1, Expr k2
where k1 = d.getAKey() and k2 = d.getAKey()
  and k1 != k2 and same_key(k1, k2)
select k1, "Duplicate key in dict literal"

When we ran this query on some test codebases, we found examples of duplicate dictionary keys. The results were also highlighted as alerts by the standard “Duplicate key in dict literal” query. For more information, see Duplicate key in dict literal.

The supporting predicate same_key checks that the keys have the same identifier. Separating this part of the logic into a supporting predicate, instead of directly including it in the query, makes it easier to understand the query as a whole. The casts defined in the predicate restrict the expression to the type specified and allow predicates to be called on the type that is cast-to. For example:

x = k1.(Num).getN()

is equivalent to

exists(Num num | num = k1 | x = num.getN())

The short version is usually used as this is easier to read.

Example finding Java-style getters¶

Returning to the example from “Functions in Python,” the query identified all methods with a single line of code and a name starting with get.

import python

from Function f
where f.getName().matches("get%") and f.isMethod()
    and count(f.getAStmt()) = 1
select f, "This function is (probably) a getter."

This basic query can be improved by checking that the one line of code is a Java-style getter of the form return self.attr.

import python

from Function f, Return ret, Attribute attr, Name self
where f.getName().matches("get%") and f.isMethod()
    and ret = f.getStmt(0) and ret.getValue() = attr
    and attr.getObject() = self and self.getId() = "self"
select f, "This function is a Java-style getter."

ret = f.getStmt(0) and ret.getValue() = attr

This condition checks that the first line in the method is a return statement and that the expression returned (ret.getValue()) is an Attribute expression. Note that the equality ret.getValue() = attr means that ret.getValue() is restricted to Attributes, since attr is an Attribute.

attr.getObject() = self and self.getId() = "self"

This condition checks that the value of the attribute (the expression to the left of the dot in value.attr) is an access to a variable called "self".

Class and function definitions¶

As Python is a dynamically typed language, class, and function definitions are executable statements. This means that a class statement is both a statement and a scope containing statements. To represent this cleanly the class definition is broken into a number of parts. At runtime, when a class definition is executed a class object is created and then assigned to a variable of the same name in the scope enclosing the class. This class is created from a code-object representing the source code for the body of the class. To represent this the ClassDef class (which represents a class statement) subclasses Assign. The Class class, which represents the body of the class, can be accessed via the ClassDef.getDefinedClass(). FunctionDef and Function are handled similarly.

Here is the relevant part of the class hierarchy:

Stmt
- Assign
  - ClassDef
  - FunctionDef
Scope
- Class
- Function