I think you're thinking of the right solution. I'll list the options.
Thinking that you would need calculations, the best solution is to not do this. Correct would be a way to normalize this. For example, if you choose decimal notation, then record 0.25 for 15min or 5 for 5h.
If you can not properly normalize and really need to know when you entered decimal or time, then the solution is to have two distinct fields and one helper indicating which one to use. Eventually you would not need this auxiliary field if you could work with null, so that which is not used would be null. Of course the application (or a restriction on the database) would have to take care to never have both used.
If this can not be done, then any solution will be bad.
However, if the field is merely descriptive and does not matter to the application what it has inside it, using a string might be appropriate.
The button to select can be a good one. Depending on the option, it would prevent the completion of the other or even change the single field shown to the user. You can change the mask. You may not even need this.
It is possible for the client application to identify which one has been typed and to prevent typing the other. You have to inform the user well that to release the other, if he wants, you have to delete one of these fields. I'm not saying it's the best solution for your users.
I can not say about the mask because I do not know what can and can not. Anyway I think this is already another question with a different problem (do separate).