In your circuit, we cannot use $i=\frac{\Delta V}{R}$ because there is no resistor in the circuit. That equation can only be used if there is a resistive element. What we have is a capacitor, so we have to use its equations. We have to use $C=\frac{Q}{V}$. Or perhaps better to avoid divide by zeros, we can reaarange it as $Q=VC$
This means that in your truly DC case, the only valid value for $Q$, the charge, is $VC$. In other words, it is impossible for the capacitor to be discharged. The question is fundamentally ill formed. The capacitor simply cannot be unchanged at t=0. There is no circuit to make sense of.
That being said, there is a related circuit that you are probably thinking of. This is a circuit with a SPDT (single pole, double throw) switch in it such that for all negative times ($t < 0$), the terminals of the capacitor are shorted together, and for all non-negative times ($t \ge 0$), the terminals of the capacitor are held at a fixed voltage by an ideal voltage source. This is not DC, it's circuit with a step function in it. But it's probably pretty close to what you were thinking.
In this case, we can analyze the circuit. At all times $t < 0$, there's 0V across the capacitor (shorting the capacitor). In this case, it is quite obvious that $Q(t)=0$ for all $t<0$, thanks to the above equation $Q=VC$. How what about at $t=0$? Well, by the way I defined the problem, at $t=0$ the switch has been flipped, so there is some $V$ across the capacitor. So by the above equation, $Q(t)=VC$ for all $t\ge 0$. This is just a true statement about ideal capacitors.
Now the infinity shows up here. Current is defined to be the flow of charge over time, $i=\frac{dQ}{dt}$. It's easy to see with the above equations that $i=0$ for all $t\ne 0$. But $t=0$ is an oddity. At $t=0$, the equation for $Q$ (charge) is not continuous. The derivative is not defined there. It's not that there's no current, its that we cannot define the current.
One intuitive approach to this is to say that $i$ was "infinite," even though that's not really true because $i$ is a real number, and "infinity" is not a real number. So for a "moment" (specifically the period of time where $t=0$, there is an "infinite" current. So if we look at charge, we get a zero divided by zero. This really just says that our approach of calling $i$ "infinite" really just didn't do the job. But we can comfortably say that immediately after the whole debacle, the capacitor must be charged to exactly $Q=CV$. No other value can satisfy the equations of an ideal capacitor.
To go further than those intuitions, we can look at a few things. One thing we can look at is a RC circuit's behavior as $R$ goes to $0$, written $\lim_{R\to 0}$. This limit is useful because limits can look at where a function is going, even if the function can never get there. Farcher's answer went that way. We can see a collection of charging curves which approach the infinite peak you see. We can see that the area under each curve is the same.
We can also use Dirac Delta Functions, as ioveri answers. Direc Deltas are interesting because the formalism behind them is intense functional analysis, but the final product is a Dirac Delta Function $\delta(t)$, which while not definable in real numbers, is astonishingly well behaved. So well behaved that it's easy to forget the formalism behind them. They work great inside integrals, and often the problems you want to use them on have an integral over time.
