@Ipeq1 Thanks for your good solution.
However, I think there is a confusing part. You consider two consecutive edges of the polygon. p0, p1, and p2 are three consecutive vertices. The vectors for the consecutive two edges should be p1-p0, and p2-p1. Since p2 and p0 are not consecutive points, p2-p0 is not an edge of the polygon if there are more than three vertices in the polygon.
@coder2 The goal is to check whether both positive and negative determinant values have been encountered when traversing through polygon vertices (ignoring all zero determinants since they are irrelevant in determining convexity).
Variable prev always stores previously last non-zero determinant. If current determinant cur is non-zero and has opposite sign from prev, we know the polygon is not convex; otherwise, cur becomes the new latest non-zero determinant and we keep looking.
The code could be written in a more readable way using two flags initialized as bool pos = false, neg = false:
if ((cur = det2(A)) > 0) pos = true;
else if (cur < 0) neg = true;
if (pos && neg) return false;
return (b-a)(c-a) - (b-a)(c-a)
d = None
for i in range(1,len(points)):
a = direction(points[i-2],points[i-1],points[i])
if a == 0: continue
if d == None: d = a
if a*d < 0: return False
if direction(points[-2],points[-1],points) * d < 0:return False
@MartyMacGyver Yes, indeed cross product is not defined for 2D space since the product vector must be orthogonal to the plane formed by the two vectors for calculation.
Another way to "embed" 2D vectors into 3D space is to force z-component as zero, i.e., let "2D" vectors
v1 = (x1, y1, 0), v2 = (x2, y2, 0),
then by the determinant formula of 3D cross product, we have
v1 x v2 = det([v1, v2, e]),
where e = (ix, iy, iz) is the triplet of 3 axis unit vectors. Only iz-direction has non-zero coefficient which is exactly 2x2 determinant det([[x1, y1], [x2, y2]]), and we only care about its sign for convexity.
@singku Good idea to calculate angles directly, but I think function call of atan is actually not really cheap and does floating point comparison of angles to 180.0 suffer numerical accuracy issues? Because even polygons with vertices of integers could have angles close to 180.0.