Actually the toString (16) method will convert to hexadecimal so from the 16. As you can see hash is a byte array and if you look at the BigInteger constructor the byte parameter [] is represented as: binary representation of magnitude of the number. You do not necessarily have to use the BigInteger class for this task. A practical example for you to better understand logic:
String message = "teste1234";
byte[] hash = MessageDigest.getInstance("MD5").digest(message.getBytes("UTF-8"));
char[] HEX_CHARS = "0123456789abcdef".toCharArray();
char[] chars = new char[2 * buf.length];
for (int i = 0; i < hash.length; ++i)
{
//Operadores bitwise para representar o valor do byte em hexadecimal
chars[2 * i] = HEX_CHARS[(buf[i] & 0xF0) >>> 4];
chars[2 * i + 1] = HEX_CHARS[buf[i] & 0x0F];
}
System.out.println("MD5: " + new String(chars));
In case the value is converted to "String" this is nothing more than the bitwise operators as shown above.
Hugs