"How can I do this?"

You can do this by using a regular expression containing the conditions you want, and testing that expression using javascript for such a shape, however you must specify such specific words or specific symbols in your regex (regular expression) so that it can validate what is typed.

The expression you initially want can be interpreted as:

var regex = /^[√\.0-9\/*\-+\()\√\π]*$/g;

Note: As you did not specify whether pi and root were to be symbolically represented or literally, I used their respective symbols .

Functional example:

textInput = document.getElementById('textInput');
textInput.addEventListener("keydown", regularexp);

buttonPi = document.getElementById('pi');
buttonPi.addEventListener("click", inserePi);

buttonRaiz = document.getElementById('raiz');
buttonRaiz.addEventListener("click", insereRaiz);

// 0-9 / * - + ( ) raiz pi [^0-9/*-+()√¯π]

function regularexp(e){
    console.log(e);
    var txt     = e.key;
    console.log(txt);
    var aryReservedKeys = [  
                             8//backspace
                           ,16//Shift
                           ,17//Ctrl
                           ,35//End
                           ,36//Home
                           ,37//ArrowLeft
                           ,38//ArrowUp
                           ,39//ArrowRight
                           ,40//ArrowDown
                           ,46//delete
                           ];
    var regex = /^[√\.0-9\/*\-+\()\√\π]*$/g;
    var isReservedCharacter = (aryReservedKeys.indexOf(e.keyCode) != -1);
    if( !regex.test(txt) && !(isReservedCharacter) ) {
        console.log("Not Match");
        var winEvent = e || window.event;
        winEvent.preventDefault();
        return false;
    }else{
        console.log("Match");
    }
}

function inserePi(){   textInput.value += 'π'; }
function insereRaiz(){ textInput.value += '√'; }

input {
    width: 100%;
}

<input id=textInput type=text value="((√31+50)-(45*55)+(5*5)/(√50*π))" />
<input id=pi type=button value="Inserir Pi" />
<input id=raiz type=button value="Inserir Raiz" />

Explanation

Note that I used a regex and added a keydown event that listens for actions on the <input id=textInput> element which checks which character was entered by the user and discards it if it is not validated by the regular expression , that is, it cancels the action.

Also, I have added some reserved actions that can eventually be used by the user as exceptions, so there is no loss of usability in your text field, as you can see such commented keys with their respective codes in the array aryReservedKeys .

Since the individual codes for each symbol are generally not known, I have created two buttons that automatically insert the pi and the root / p>

Adding more symbols / words

Good to add a symbol is extremely simple, you should just add after the last symbol validated by the regex a backslash and then the desired symbol such as: \% So the new regex would be arranged this way:

var regex = /^[√\.0-9\/*\-+\()\√\π\%]*$/g;

Now for words, just write them exactly as they are in parentheses then getting in this way, for example, let's add the word wordTest :

var regex = /^[√\.0-9\/*\-+\()\√\π*]*(palavraTeste)*$/g;

Extra

To find out which regular expressions are correct for what you want to do, and to understand how the regular expression works, use this site as reference, it is very good.

To allow this 0-9 / * - + ( ) raiz pi would be something like [0-9/\*\+\-\(\)\.] .

For specific words it would look something like this: (palavraUm|palavraDois|palavraTres)

I suggest you read this link to better understand how it works.

In this link has videos tutorials in Portuguese.