It seems that Eclipse will try to adjust the property page dialog according to the property page itself by calling the computeSize method. As some control in the property page do not explicitly override the computeSize method, such calling of computeSize method will go the layout's computeSize, the layout will always return a big size for the SWT.DEFAULT parameter, so the whole property page will be much bigger than normal.

Try to override the computeSize method, such as:
    public Point computeSize(int wHint, int hHint, boolean changed) {
        Point point = super.computeSize(wHint, hHint, changed);
        if (point.x > 500 && wHint == SWT.DEFAULT) {
            point.x = 500;
        }
        if (point.y > 400 && hHint == SWT.DEFAULT) {
            point.y = 400;
        }
        return point;
    }