Surveys

The surveys you see on television and in the newspapers are suspicious. They can be accurate, but still wrong.
For fifteen or twenty years I worked with scientists; Social scientists. Sociologists, psychologists, and statisticians, highly educated people holding doctorates. Dave had a doctorate and two master’s. His doctorate was in statistics, as were a few others’.
I learned a lot from my learned colleagues; I was nowhere near as educated, having only spent four years at a university, and that as an art major. I started out entering data and wound up programming the PC databases (Dave and the other statisticians used SAS, a very powerful and expensive mainframe statistical analysis program).
Many times they did surveys of the poor to gauge the effectiveness of the programs we were testing. I don’t know but a fraction what they, especially the statisticians, knew about surveys, but I now know a lot more than most laymen. I even learned 101 level statistics myself; Dave had written a textbook on the subject that was used in colleges, and I read it.
First, telephone surveys used to be very accurate, back in the day before answering machines were invented, long before cell phones. Your phone was tethered to a wall in your house, and when it rang, you answered it. It was almost always a call you wanted to take, as before the internet and automated phone dialers the only spam was physical junk mail.
Then around 1980 those automated phone dialers came about. It spawned the invention of its solution: the answering machine. The phone spam problem wasn’t yet widespread enough to be a problem, but was for many municipalities.
People mostly still answered their phones, but if you were eating dinner you would just call them back. If the message was from a pollster, there was less chance of them getting your view, although it didn’t yet affect outcomes in a statistical manner.
Then when everyone had a smart phone in their pocket the spam problem skyrocketed. Everyone also had caller ID and their personal phone books, as well as voicemail. Slowly people stopped answering numbers and only answered calls with names attached.
At this point, telephone surveys became no more reliable than wild guesses, since few people would answer a number. Answers were skewed because only a certain subset of the population answered, mostly the elderly who still maintained landline phones.
Second, about opt-in surveys: all are garbage. Period. By “opt-in” I mean you’re not a targeted subject, as a list of phone numbers a call center would call, or printed questionnaires mailed out to the target population. If it’s open to everyone, only those with a strong opinion will answer. The data are meaningless.
A survey about food sent only to vegans, for example, needs no leading questions for the survey to say that people are against eating meat. To survey vegans and not tell anyone that vegans were targeted would be fraudulent, but it happens. Donald Trump wasn’t the world’s first fraudster, just the most successful.
How a question is asked has a huge bearing on how it will be answered. For instance, “should the government spend more than it has?” and “should the debt limit be raised?” are two different questions, but reporters may conflate them. A good survey will ask the same question in different ways.
People lie on surveys despite wanting their opinion known, and I don’t understand why but I’ve seen it. With one survey we sent out, almost all of the negative responses had the document number torn off, the respondent obviously thinking they were being tracked. Oh, you were not being tracked, those were document numbers, identical on every one..
All political party surveys mailed to you are bogus, and an excellent example of this type of fraud. As I’ve voted in both parties’ primaries, both send me junk mail. In Illinois, the Republicans are especially bad about that these days but both are guilty.
If you’ve been sent a legitimate survey rather than one that was produced for propaganda purposes, you may wonder why it seems like “didn’t I just answer this a few questions ago?” The reason is that they’re looking for those who are just checking stuff off without actually paying attention to what is asked. You’re not likely to fool a psychologist who sends you a survey.
It takes a team to produce a useful survey. Unless a survey is designed by people who know statistics and psychology its findings are garbage and it’s worthless. It would be like letting an astronomer remove your appendix. Yes, he’s a doctor, but not in that field.
1/1/2024