I do not want to be pessimistic, but it's almost impossible to know / prevent executions of screen capture software, you can create a huge list of known capture software and make your software access the list of running windows processes , you'll simply compare which process hits your list of "unwanted" capture software.
Or have your software access the list of applications installed on windows and see if any names match your known list. if you have a positive response it is up to you to alert the user or simply not to play the video.
To completely bar things like this you would have to practically rewrite the video driver, ie literally bar the source!
Remember that you still run the risk of the guy simply picking up a cell phone / camera / camcorder and recording.
To minimize you can stamp your videos with a watermark, if the video will leak it will have the logo of your company, nowadays I know that some companies render brands at the time of the stream with subtle modifications for each user, the naked eye all the marks look the same, but when applied fingerprint methods in the leaked video it is possible to trace by that mark which user was responsible.
You can try to access windows hot keys by your application and try to disable the screen capture keys, see an example of how to register and disable the capture of print screen using C # here , if some capture software uses this same hot key to capture will work more, but unfortunately you will have little success, the vast majority of capture programs define their own hot keys is almost like shooting in the dark.
