In the first 2 and last 5 hours, Governors are free to move across servers and battle. During the 5 hours in between, you may attack the King's Castle.