Plotly: Plot Polygons

The problem is in the order of your points. I have made an example with the same points but organized them differently (different positions in the array).

fig = make_subplots(1,3, subplot_titles=("polygon0", "polygon1", "polygon2"))

X0 = [25, 50, 38, 40]
Y0 = [20, 40, 60, 25]

X1 = [25, 38, 50, 40]
Y1 = [20, 60, 40, 25]

X2 = [25, 38, 50, 40, 25]
Y2 = [20, 60, 40, 25, 20]


polygon0 = go.Scatter(
        x=X0,
        y=Y0,
        showlegend=False,
        mode="lines",
        fill='toself',
        line=dict(color="LightSeaGreen", width=2)
        )  

polygon1 = go.Scatter(
        x=X1,
        y=Y1,
        showlegend=False,
        mode="lines",
        fill='toself',
        line=dict(color="LightSeaGreen", width=2)
        )  

polygon2 = go.Scatter(
        x=X2,
        y=Y2,
        showlegend=False,
        mode="lines",
        fill='toself',
        line=dict(color="LightSeaGreen", width=2)
        )  

fig.add_trace(polygon0, row=1, col=1)
fig.add_trace(polygon1, row=1, col=2)
fig.add_trace(polygon2, row=1, col=3)
fig.update_xaxes(range=[20, 55])
fig.show()

As you can see in the figure polygon 0 and polygon 1 are very different althought are constructed from the same scatter points. The only difference in the arrays is that points 2-(50, 40) and 3-(38, 60) are changed in the array X1 Y1, (2 becomes 3 and 3 becomes 2).

Finally, if you want to close your polygon, you can add an extra point to your arrays equivalent to point 0 (polygon2).

Adding to answer from @Tengis Using shapely you can construct a multipoint geometry and from that the convex hull. That's then simple to use in plotly

from plotly.subplots import make_subplots
import plotly.graph_objects as go
import shapely.geometry
import numpy as np

fig = make_subplots(1, 2, subplot_titles=("polygon0", "convex hull"))

X0 = [25, 50, 38, 40]
Y0 = [20, 40, 60, 25]

# 2D array of points that make up convex hull
convex_hull = np.array(
    shapely.geometry.MultiPoint(
        [xy for xy in zip(X0, Y0)]
    ).convex_hull.exterior.coords
)

polygon0 = go.Scatter(
    x=X0,
    y=Y0,
    showlegend=False,
    mode="lines",
    fill="toself",
    line=dict(color="LightSeaGreen", width=2),
)

polygon1 = go.Scatter(
    x=convex_hull[:, 0],
    y=convex_hull[:, 1],
    showlegend=False,
    mode="lines",
    fill="toself",
    line=dict(color="LightSeaGreen", width=2),
)

fig.add_trace(polygon0, row=1, col=1)
fig.add_trace(polygon1, row=1, col=2)
fig.update_xaxes(range=[20, 55])
fig.show()