Tough times for Whyte