What Evidence is Informal Internet Surveys?

One of my primary ways of learning about this gender stuff is by posting self-made surveys to various places, usually /r/SampleSize. Inevitably, this leads to people complaining that my surveys aren’t “real science” and should therefore be disregarded. But is that fair?

The main problem with this is that the way to learn about things isn’t to require that all information must meet some arbitrary standard of quality before you update your beliefs. All data must be taken into consideration for maximal accuracy, not just the best data. Since we often don’t have data in high quality, this is a very important thing to note, since otherwise you will get stuck with…

With what? What do people do if they don’t use my surveys? Usually, they seem to go with vague anecdotes, personal experiences, and their own political priors. This approach is hardly more accurate than informal internet surveys. It’s much more convenient, of course, because it lets you make up whatever you need for your politics. This is a great way to win, but it’s not going to put you any closer to the truth.

Of course, if there’s quality scientific evidence on some question, then I’m fully willing to use this in addition to my surveys. However, it’s worth noting that most cases I’ve seen people cite science that supposedly contradicted my findings, it turned out to mostly agree. For example, I was once debating with someone who argued that women hate their sex organs, so I mentioned that this contradicted what I’d found in my surveys. She then cited several papers which found… that women are ambivalent-or-happy about them. Which is exactly the same as I found.

There’s also the problem where scientific evidence isn’t necessarily as good as people assume. Take for example Littman’s study of ROGD. I’m sure it follows all the best standards of science and has been made by experienced people. However, Littman never gets data directly from the actual subjects she’s trying to study, and instead collects it from parents that have been recruited on websites that are biased against the claims of the subjects. She doesn’t examine highly relevant factors like AAP, and she doesn’t seem to even consider that the places she’s recruited people might caused bias.

There are all sorts of things that could go wrong with my surveys. There are also all sorts of things that could go wrong with actual science, though perhaps a bit fewer. On the other hand, if you get rid of my surveys, you’re not going to get science, you’re going to get baseless speculation, which is much worse. So I say, let’s do some surveys!

You’re not going to be satisfied with that, right? The followup that always comes after this is that maybe my surveys do provide more accurate information than wild speculation, but the fact that this information is numbers means that we need to hold them to higher standards, because people will respect numbers much more than wild speculation.

It’s possible that this is the case, but then we need to change how people treat numbers, not whether we use my surveys. Numbers are not this magical unquestionable substance, they’re just a useful way to summarize information, and it’s insane to avoid them.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s