Is there any way to make drawn small objects in OpenGL in Java not so jagged?

I'm trying to smooth the edges of a rendered object. It looks like this:

The edges of this object are not smooth.

This object is drawn in game overlay using method that uses custom renderer and framebuffers: WorldRenderer and Tesselator, but I don't know if having this knowledge is needed for answering this question.

I have tried adding this lines of code, but it didn't work:

GL11.glTexParameteri(GL11.GL_TEXTURE_2D, GL11.GL_TEXTURE_MIN_FILTER, GL_LINEAR);
GL11.glTexParameteri(GL11.GL_TEXTURE_2D, GL11.GL_TEXTURE_MAG_FILTER, GL_LINEAR);
GL11.glDisable(GL_DEPTH_TEST);

Drawing object method:

    public static void drawCircle(final double x, final double y, final double r, final int c) {
        glEnable(GL_BLEND);
        glDisable(GL_TEXTURE_2D);
        glEnable(GL_POLYGON_SMOOTH);
        glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);
        glHint(GL_POLYGON_SMOOTH_HINT, GL_NICEST);
        GLUtils.glColor(c);

        glBegin(GL_POLYGON);
        for (int i = 0; i <= 360; i++) {
            final double x2 = Math.sin(((i * Math.PI) / 180)) * r;
            final double y2 = Math.cos(((i * Math.PI) / 180)) * r;
            glVertex3d(x + x2, y + y2, 0);
        }

        glEnd();

        glDisable(GL_POLYGON_SMOOTH);
        glEnable(GL_TEXTURE_2D);
    }

I also noticed that if I render rectangle or something else behind the object (in same method, not from seperate classes), the edges aren't pixelated, but I want to smooth edges without doing it:

If you need something more, let me know. Thank you very much for help.