In the first 2 and last 4 hours, Governors are free to move across servers and battle. During the 6 hours in between, you may attack the King's Castle.