Instantly, you can think of an O(N) solution.

When given intervals, for example (a,b) and (c,d) where d > b > c, we know that there will be no free interval between a and d. If all the intervals are sorted by its start time, we could just iterate through the array of intervals to find all (s,e) such that there are (x,s) and (e,y) such that s <= e.