On 'going out for a smoke'

I am working most of the time, and I am very busy. So when I go out, it's typically an outing I've organized. When I get asked "Wanna go out for a smoke?", I would usually go out just to be good company but now I don't do that. Now people ask me "Wanna go out for a smoke?" and since it's an outing I've arranged, the thing we're going out for is something I want to be a part of so my answer more and more is "No, I don't want to go out for a smoke." Going out for a smoke makes you change gears, and it almost seems like it's a release valve to make sure one doesn't have too much fun. It always feels like the mood of the situation is just getting good when someone wants to go out for a smoke. And when I don't go, I always get a look like it's a surprise or like I'm a jerk. I've always said that a smoking habit is a thing that keeps you from thinking too hard because your thoughts will only go so far before something makes you think of a smoke, and you go "that's a good idea!", and it's smoke time again. If something's important enough to interupt my work for, I'm interested enough in it to pay it my full attention, not five minutes worth at a time.