I also wondered about this back when I was building my bus dashboard. I got around half a volt between the USB GND and my 24VDC power supply. My issue wasn't with this, that means almost nothing. But I was afraid what if something goes wrong on the power supply part, that could have an impact on my PC motherboard (sure that's got a very little chance and USB buses have fuses built in, which should limit the current to 500mA, but what if...).
Anyway the simpliest method if you use optocouplers. They are relatively cheap (one IC is maybe 0.5-1€ maximum) and easy to use (they have 2 input and 2 output - the inputs need to be hooked up like an LED, since they actually are, and the outputs can be used as switches). That way you have your Arduino coupled to your PC via GND (and the rectified 5V and 3,3V too, which come from the USB 5V pin) and their outputs are used to switch the 24VDC power supply stuff. There's almost no way it will ever have any impact on your PC, only at very high voltage (since there's no physical contact between the sides).
Sure it's maybe a bit overkill, but that's the safest and easiest way I can imagine of, and I would never harm my motherboard because of a wrongly hooked up cable or anything like that...
Edit:
Just a few things came to my mind. Power supplies don't always have their ground hooked up to "real ground" through power network. Just imagine a simple battery, its terminals are floating, they are initially not connected to anywhere. But some power supplies have their negative pin hooked up to GND. That doesn't mean it's the same potential as the GND pin on USB. The reason is the wire resistance of the components, that's the reason for the little voltage difference between the GND pins. If you would measure the voltage between the GND of the USB and let's say a grounded metal pipe in your house, there also would be a little difference for the same reason.
Arduinos don't like high back current (back EMF) from loads like coils (motors or even like relais). In needed cases you can just hook up a diode to output this risk (which will conduct the back current, but have no impact on the circuit at normal states).
By the way you can even use optocouplers at PWM or "analog" outputs. The cut-off frequency of the optocouplers are in the kHz spectrum, but that's not an issue. If you have let's say a gauge which need analog voltage between 0-24V, that will just work as a very crappy low pass filter (the DC part still "gets through"), and due to the "high" frequency of the PWM signal and the physical inertia of the moving parts of the gauge you won't see the needle shaking.. Also if you have a gauge which works according to the frequency of it's signal (engine rev meters eg.), it's still not an issue, because they usually need a few 10kHz maybe (it depends on the meter of course) and usually those frequencies are not cut to zero.