The following diagram should help:
We know of course the relationship between $\alpha$ and $\beta$ because the angle $A$ is known: $$\alpha + \beta = A$$
Next, we have the relationship between $i$ and $\alpha$ from Snell's law:
$$\frac{\sin{i}}{n}=\sin\alpha$$
and similarly for $\beta$:
$$\frac{\sin{e}}{n}=\sin\beta$$
This gives you three equations with three unknowns: $\alpha,~\beta,~n$.
I would recommend solving this numerically, or graphically. Simple example of a graphical solution:
I calculated the expected exit angle given an input angle of 45° and a refractive index of 1.543 - then calculated what exit angle I would get if I didn't know the refractive index. The error goes to zero when you have the right index...
Python code used to generate this:
# prism simulation
from math import pi, asin, sin
import numpy as np
import matplotlib.pyplot as plt
A = 60*pi/180. # 60 degrees, in radians
# pick an angle of incidence
i = pi/4
# pick a refractive index
n = 1.543
# compute the exit angle
alpha = asin(sin(i)/n)
beta = A - alpha
e = asin(sin(beta)*n)
print 'exit angle is %.1f deg'%(e*180/pi)
# calculate n graphically from just i and e
na = np.arange(1.4,1.6,0.001) # possible values
# what would the intermediate values alpha, beta be?
aa = np.arcsin(np.sin(i)/na)
bb = A - aa
ee = np.arcsin(np.sin(bb)*na)
plt.figure()
plt.plot(na, 180*(ee-e)/pi)
plt.plot([n,n],[-20,20])
plt.plot([1.4,1.6],[0,0])
plt.xlabel('possible value of n')
plt.ylabel('error in exit angle')
plt.title('graphically estimating n')
plt.show()