I'm confused for this problem that we choose median to be the number everyone tries to move to, but not the average. Some post gives the reason that if we choose a number in the array, we save operations to change itself. But I'm thinking this reason doesn't stand because if we choose the average, other numbers may take less moves than choosing median.

Can anyone help me understand the thinking process of choosing median instead of average?

Thanks!