I've suffered anxiety/panic attacks 10 years, I tried a little test, On Jan 1st 2007 I went of both Caffeine and Alcohol, I can honestly hand on heart say I felt no difference what so ever. In saying that I was never addicted to caffeine anyway, I could drink maybe 4 cups of coffee 1 day and have none the next and have absolutly no desire for it.
I started back on the caffeine start of April, It was a bottle of Jolt cola and we all know the amount of caffeine that stuff has in it, I didn't notice 1 bit of difference after drinking it and that was me without 1 bit of Caffeine in 3 months.