Your team’s 4 highest-level skills will only be recalculated when the allies that provide the said skills leave. The departure of other allies will not affect the current skill effects.