You know, I had problems just like this BEFORE I became a vegetarian, then I became a veggie and started doing daily cardio. I had panic disorder and after the changes I made eventually worked through it to a livable level. While I do believe that meat intake is different from person to person, don't be so quick to assume not eating meat has anything to do with it at all. It's the same way some people can take a shit load of sleeping pills and not get tired (like myself), and some people can overdose on one. It's more likely there are other issues at hand then a meat free diet. However, like whats been said, unless you are looking to loose weight, you want to have enough calories in you to function properly, because you burn then off all day and it isn't really hard to do. But don't throw a crap load of doritos or cupcakes in you because they are high in calories, your body won't function well off of those. If you aren't eating right, or at least decent enough for what your body personally needs you can become irritable and even depressed, which would 'cause high anxiety and loss of appetite, high anxiety can force your heart to beat to fast which triggers the fight or flight response. If your body chooses flight, then you are probably going to experience something along the lines of being light headed, which if you are unaware as to why it's happening will cause fear, which of course will cause more anxiety and make it all worse (this is why people who experience a panic attack the first time believe they are dying). This can become a neurological problem.
In short, take care of yourself, figure out what is wrong with you and then you can figure out how to combat the experience you are having. The only way to do this is to analyse your situation when the events take place, or preferably after you have gotten yourself feeling better again, after all you don't want to make it worse.
How much have you been drinking? These could also be after affects of alcohol.
Also, your body needs rest if you aren't getting enough sleep, this too can become a problem with the same side affects.