I think your two pointer method is correct, but the explanation is a little ambiguous, it is like Greedy idea, always choose small value to forward/backward. I think this problem can be converted to Elevator schedule problem which wants find out which floor the elevator stops can make the total steps people need to take to their floor minimum.

let A[i] is the total steps if elevator stop ith floor, then there are following transition equation:

A[i] = A[i-1] + sum[0,i-1]-sum[i,n-1], we need to find the flecxion point for sum[0,i-1]-sum[i,n-1], i is in [0,n-1]. This flecxion point is the floor or meeting point.

By this idea, I give following O(n) solution for finding meeting point.

// the simple case to get the median number for every floor has 1 person at most.
private int minMoves1D(int[] nums) {
int sum = 0;
for (int num: nums) sum += num;
if (sum == 0) return 0;
int pos = 0, cur = 0;
while (pos < nums.length && cur < sum) {
cur += nums[pos];
sum -= nums[pos];
pos++;
}
pos--;
int res = 0;
for (int i = 0; i < nums.length; i++) res += nums[i]*Math.abs(pos-i);
return res;
}