It could just be a mechanical issue or a mentality issue. It could be that throwing with a baseball makes you feel like you can't move as fast down the mound, stuffing your mechanics. Yes and like Brent said, throwing a baseball is nothing like using a towel. Try doing it slowly first and getting it right instead of always trying to speed your way down the mound, then work your way up.