I do not think you understood how preg_replace work because you passed as second parameter a regular expression, when should use the text that should replace what was found.
In addition you are escaping (with backslash) the >
character that has no special function in the regular expression.
To make your code work you should use a subpatern (group delimited with parentheses) and references ($ n where n is a number):
preg_replace('/<li>(\s*<p>[a-z]\)\s)/', '<li class="inciso">$1', $documento);
Explanation of this regular expression can be found in this answer