Physical difference between $\vert S=0, m = 0 \rangle$ and $\vert S=1, m = 0 \rangle$? In context of a two spin $\frac{1}{2}$ particle systems, we know that,
$\vert S=0, m = 0 \rangle = \frac{1}{\sqrt2}(\vert\uparrow\downarrow\rangle - \vert\downarrow\uparrow\rangle)$
$\vert S=1, m = 0 \rangle = \frac{1}{\sqrt2}(\vert\uparrow\downarrow\rangle + \vert\downarrow\uparrow\rangle) $
I understand they are orthogonal for states being orthonormal and antisymmetic for satisfying the exclusion principle.
But can I not flip the positive and negative signs and still have orthogonal states?
The best answer I could find used the spin ladder operators to act on the states. Looking for a more "physical" answer if it exists.
 A: 
In context of a two spin $\frac{1}{2}$ particle systems, we know that,


$\vert S=0, m = 0 \rangle = \frac{1}{\sqrt2}(\vert\uparrow\downarrow\rangle - \vert\downarrow\uparrow\rangle)$


$\vert S=1, m = 0 \rangle = \frac{1}{\sqrt2}(\vert\uparrow\downarrow\rangle + \vert\downarrow\uparrow\rangle) $


I understand they are orthogonal for states being orthonormal and antisymmetic for satisfying the exclusion principle.

They or orthogonal to each other because:
$$
<S=0,m=0|S=1,m=0> = 0
$$
The rest of what you wrote ("for states being orthonormal and antisymmetic for satisfying the exclusion principle") doesn't make sense to me, or at least it doesn't make sense why you wrote it. You seem to be justifying their orthogonality with this sentence fragment. But the states are just orthogonal, full stop.

But can I not flip the positive and negative signs and still have orthogonal states?

Which sign? You can flip the overall sign or add any overall phase and they states will still be orthogonal. If you flip the middle sign then you change one state into the other.

The best answer I could find used the spin ladder operators to act on the states. Looking for a more "physical" answer if it exists.

The "physical" answer is that the |S=1,m=0> state has a finite angular momentum given by $\hbar\sqrt{2}$, and the |S=0,m=0> state has zero angular momentum.

A little more discussion might help you feel better about these states.
First of all, no one is forcing you to work with these
$|S,m>$ states as your basis (ok, maybe your teacher is forcing you, but you get what I am saying). If you prefer, you could work in a different basis, such as:
$$
|1> = \vert\downarrow\downarrow\rangle
$$
$$
|2> = \vert\downarrow\uparrow\rangle
$$
$$
|3> = \vert\uparrow\downarrow\rangle
$$
$$
|4> = \vert\uparrow\uparrow\rangle
$$
Second, in a lot of problems, you are often interested in operators that commute with the total spin squared and total spin along one axis. Then, to solve the problem you usually have to diagonalize a matrix representation of the operator.
Third, if you are already working in a basis like $|S,m>$ some of the work is already done for you. So, this is why we often like to work in this basis.

As an example of how to work directly with the direct product basis, consider:
$$
\hat S_z |\downarrow\uparrow\rangle = \left(s_z^{(1)}\times {I}_2 + {I}_2\times s_z^{(2)}\right)|\downarrow\uparrow\rangle = \frac{-1}{2}|\downarrow\uparrow\rangle
+\frac{1}{2}|\downarrow\uparrow\rangle = 0
$$
$$
\hat S_z |\uparrow\downarrow\rangle = \left(s_z^{(1)}\times {I}_2 + {I}_2\times s_z^{(2)}\right)|\uparrow\downarrow\rangle = \frac{1}{2}|\uparrow\downarrow\rangle
+\frac{-1}{2}|\uparrow\downarrow\rangle = 0
$$
$$
\hat S_x |\downarrow\uparrow\rangle = \left(s_x^{(1)}\times {I}_2 + {I}_2\times s_x^{(2)}\right)|\downarrow\uparrow\rangle = \frac{1}{2}|\uparrow\uparrow\rangle
+\frac{1}{2}|\downarrow\downarrow\rangle
$$
$$
\hat S_x |\uparrow\downarrow\rangle = \left(s_x^{(1)}\times {I}_2 + {I}_2\times s_x^{(2)}\right)|\uparrow\downarrow\rangle = \frac{1}{2}|\downarrow\downarrow\rangle
+\frac{-1}{2}|\uparrow\uparrow\rangle
$$
$$
\hat S_y |\downarrow\uparrow\rangle = \left(s_y^{(1)}\times {I}_2 + {I}_2\times s_y^{(2)}\right)|\downarrow\uparrow\rangle = \frac{-i}{2}|\uparrow\uparrow\rangle
+\frac{i}{2}|\downarrow\downarrow\rangle
$$
$$
\hat S_y |\uparrow\downarrow\rangle = \left(s_y^{(1)}\times {I}_2 + {I}_2\times s_y^{(2)}\right)|\uparrow\downarrow\rangle = \frac{i}{2}|\downarrow\downarrow\rangle
+\frac{-i}{2}|\uparrow\uparrow\rangle
$$
You can similarly find the action of $S_i$ on the other two direct product basis states: $|\uparrow \uparrow\rangle$ and $|\downarrow \downarrow\rangle$.
From here you should be able to explicitly construct:
$$
\hat S^2 = \hat S_z^2 + \hat S_x^2 + \hat S_y^2
$$
as a 4x4 matrix and show its action on the two states of interest.
The result is:
$$
\hat S^2 = \begin{bmatrix}2 & 0 & 0 & 0
\\0 & 1 & 1 & 0
\\0 & 1 & 1 & 0
\\0 & 0 & 0 & 2
\end{bmatrix}\;,
$$
where (surprise surprise) the eigenvalues are: 2, 2, 2, 0.
The three eigenvectors with eigenvalue 2 make up the S=1 space. The one eigenvector with eigenvalue 0 is the S=0 space.
Your original question about flipping the sign should also be clearly answered via this explicit construction as well. The vector with the relative minus sign clearly has $\hat S^2 = 0$ and the vector without the relative minus sign clearly has $\hat S^2 = 2$. (Note that the eigenvalue of the operator $S^2$ is called $S(S+1)$ where $S$ is the spin.)
