@coder2 The goal is to check whether both positive and negative determinant values have been encountered when traversing through polygon vertices (ignoring all zero determinants since they are irrelevant in determining convexity).
Variable prev always stores previously last non-zero determinant. If current determinant cur is non-zero and has opposite sign from prev, we know the polygon is not convex; otherwise, cur becomes the new latest non-zero determinant and we keep looking.
The code could be written in a more readable way using two flags initialized as bool pos = false, neg = false:
if ((cur = det2(A)) > 0) pos = true;
else if (cur < 0) neg = true;
if (pos && neg) return false;
@Ipeq1 Thanks for your good solution.
However, I think there is a confusing part. You consider two consecutive edges of the polygon. p0, p1, and p2 are three consecutive vertices. The vectors for the consecutive two edges should be p1-p0, and p2-p1. Since p2 and p0 are not consecutive points, p2-p0 is not an edge of the polygon if there are more than three vertices in the polygon.
return (b-a)(c-a) - (b-a)(c-a)
d = None
for i in range(1,len(points)):
a = direction(points[i-2],points[i-1],points[i])
if a == 0: continue
if d == None: d = a
if a*d < 0: return False
if direction(points[-2],points[-1],points) * d < 0:return False