TL; DR - Nothing to worry about.
Long answer :
The risks involved in disconnecting the USB port during debugging are exactly the same as what the device is subjected to during everyday use.
In some older USB implementations it was possible to remove a device while the host is still delivering power to the device. This is very rare today.
The main problem with non-eject removal is the fact that some operating systems cache memory in files being handled on removable media. As in practice you are not manipulating files (ADB takes care of 'serializing' and sending your APKs using your own protocol), this risk is non-existent.
On the device side, the operating environment ensures that your new APK package will only be installed if the transfer occurs successfully.