Let me just say I really don't notice a whole lot of difference, but I have people who do have better sense of sound who find the optical jack much better. I do 'imagine' highs and lows sounding better when I know an optical connection is being made, but I really can't tell if doing a blind test. I do notice the difference when using CD's, dvd's or blueray because I think the quality of the recording is better and more direct as opposed to being streamed by wifi.
There is no difference in RCA and 3.5mm they are both analog. Thicker cable on both over a long distance might prevent potential interference. But Optical is a 'cleaner' signal and digital with more information.