I remember that this is a question from the "Cracking the coding interview" book.

The idea of the original solution is to first find an index from left that the corresponding array value breaks the ascending order, another index from right that breaks the descending order.

Then in the subarray between these two indices, find the min and max values. Back to the array, find the position from left that large than the min value, and the position from right that smaller than the max value, the distance of these two positions will be the answer.

Here is my implementation, though messy, just for reference.

class Solution {
public int findUnsortedSubarray(int[] nums) {
if (nums == null || nums.length < 2) {
return 0;
}
if (nums[0] > nums[nums.length-1]) {
return nums.length;
}
int left = -1;
int right = -1;
for (int i = 1; i < nums.length; ++i) {
if (nums[i-1] > nums[i]) {
left = i - 1;
break;
}
}
if (left == -1) {
return 0;
}
for (int i = nums.length - 2; i >= 0; --i) {
if (nums[i] > nums[i+1]) {
right = i + 1;
break;
}
}
int start = Math.min(left, right);
int end = Math.max(left, right);
int min = Integer.MAX_VALUE;
int max = Integer.MIN_VALUE;
for (int i = start; i <= end; ++i) {
min = Math.min(min, nums[i]);
max = Math.max(max, nums[i]);
}
for (int i = 0; i < nums.length; ++i) {
if (nums[i] > min) {
start = i;
break;
}
}
for (int i = nums.length - 1; i >= 0; --i) {
if (nums[i] < max) {
end = i;
break;
}
}
return end - start + 1;
}
}