Bad HTML filtering regexp¶
ID: py/bad-tag-filter
Kind: problem
Security severity: 7.8
Severity: warning
Precision: high
Tags:
- correctness
- security
- external/cwe/cwe-116
- external/cwe/cwe-020
- external/cwe/cwe-185
- external/cwe/cwe-186
Query suites:
- python-code-scanning.qls
- python-security-extended.qls
- python-security-and-quality.qls
Click to see the query in the CodeQL repository
It is possible to match some single HTML tags using regular expressions (parsing general HTML using regular expressions is impossible). However, if the regular expression is not written well it might be possible to circumvent it, which can lead to cross-site scripting or other security issues.
Some of these mistakes are caused by browsers having very forgiving HTML parsers, and will often render invalid HTML containing syntax errors. Regular expressions that attempt to match HTML should also recognize tags containing such syntax errors.
Recommendation¶
Use a well-tested sanitization or parser library if at all possible. These libraries are much more likely to handle corner cases correctly than a custom implementation.
Example¶
The following example attempts to filters out all <script>
tags.
import re
def filterScriptTags(content):
oldContent = ""
while oldContent != content:
oldContent = content
content = re.sub(r'<script.*?>.*?</script>', '', content, flags= re.DOTALL | re.IGNORECASE)
return content
The above sanitizer does not filter out all <script>
tags. Browsers will not only accept </script>
as script end tags, but also tags such as </script foo="bar">
even though it is a parser error. This means that an attack string such as <script>alert(1)</script foo="bar">
will not be filtered by the function, and alert(1)
will be executed by a browser if the string is rendered as HTML.
Other corner cases include that HTML comments can end with --!>
, and that HTML tag names can contain upper case characters.
References¶
Securitum: The Curious Case of Copy & Paste.
stackoverflow.com: You can’t parse [X]HTML with regex.
HTML Standard: Comment end bang state.
stackoverflow.com: Why aren’t browsers strict about HTML?.
Common Weakness Enumeration: CWE-116.
Common Weakness Enumeration: CWE-20.
Common Weakness Enumeration: CWE-185.
Common Weakness Enumeration: CWE-186.