I've also heard of the weed causing allergic reactions, but it is not different than any other allergy... some people have it worse than others, but the second hand smoke of any plant doesn't cause people to react. Weed pollen gets in the air like other plants, so you'd have to handle the plant or pick the fruit to have a reaction. Smoking it also causes allergic reactions to some (these can be the most severe reactions); I could be wrong or misinformed but I haven't come across any research to think a whiff would trigger a reaction...
I also wanted to mention, though some unfortunately feel disrespected by those who smoke in parks, many neighbours feel the same about those smoking at home. There in lies a dilemma.
The replies in this thread are helpful in this manner to gauge the vibe of the birders' views and how to maximize consideration when birding.
The irony for myself and some of my friends is that smoking the weed outdoors is what led to the curiosity of these peculiar and fascinating creatures called birds. I can now see why hippies love nature so much! Appreciation breeds understanding. It seems to me that hippies are environmentalists for the same reasons birders are: appreciation!
...On a different note, I'd really don't appreciate those cedar waxwings always drinking in our parks. Some of them even drink and fly!!