From time to time I'd have a lot of difficulty starting, and that's why I used the drill. Sometimes there would be a mismatch between speeds when the motor fired it felt for a second like it was going backwards. That's where I think the extra stress came in... basically using the drill to try to stop the flywheel (intermittently for a very short time), so that stress would presumably be felt at the junction of flywheel and crank. That's my theory anyway, doesn't explain yours!
The only info I have on flywheel weight is not to use anything much more than 1.6kg with a Mazzuchelli bell crank, but I think you were staying with rotary?