Generalization Arguments                                                                                                                                     (Problems printing? Click here.)

If you wish to download a copy to read off-line, I suggest you click on ctl_general_basics.rtf, and download a smaller and more easily printable rtf file. It should fit on 22 pages, and includes a seperate homework sheet.

Sometimes we want to be able to figure out the truth or falsity of general statements. These are statements that cover the whole of some population, such as Americans, wombats, the water in the oceans, left-handed Armenian mole-diggers, Scotsmen with Irish names, tea-drinkers, trees, people who do horrible things to turnips ... well, you get the idea. Any time we want to claim that some statement about all of something is true, we have to support that claim with a generalization argument.

Sample

The essence of a generalization argument is the "sample." Usually, populations are so large that we cannot reasonably test the state of every member of that population. For instance, if we wanted to know what proportion of Scotsmen get tipsy (slightly drunk) on Hogmanay, we cannot possibly hire enough obervers to follow around every Scotsman around on the evening of December 31st. (Especially if we count female Scots as "Scotsmen" Oh, lets just call them "Scots."), so we're scre... I mean, so we have to fall back on looking at a much smaller number of Scots and extrapolating the results to all haggis-eatin', kilt-wearin' caber-tossers. (This is perhaps an unfair characterization of the Scots. Very few of them actually toss cabers.) So let's just hire people to follow around a randomly selected group of one million Scots next Hogmanay and to report on whether or not they get tipsy. Say that 75% of these randomly selected Scots get tipsy on Hogmanay, we could then make the following argument.

Exactly 75% of our sample got tipsy this Hogmanay, therefore 75% of all haggis-eaters got tipsy this Hogmanay.

Here's how the terminology of generalization matches up with this argument.

Facts
Population: All Scots (several million of them.)
Sample: One million randomly selected Scots.
Property being tested: Tipsiness.
State of the sample: 75% tipsy at this Hogmanay.

Conclusion drawn from those facts
State of the population : 75% tipsy at this Hogmanay.

This is how a generalization works, if it works at all. A sample is taken, and it is argued that the state of the sample must be the same as the state of the population. If the state of the sample cannot reasonably be explained without assuming that the population has the same state, the argument is good. If we can reasonably explain the state of the sample without assuming that the population has the same state, the argument is no good, lousy, bogus, wack, heinous .... I'll stop now.

For another example, imagine that two people, call them "Jeeves" and "Wooster," are trying to figure out the overall composition of the following population. Imagine also that neither of them can see the population the way you can. (You can see that this population is extremely well mixed. In fact, there are only two deviations from perfect mixing. They appear in the top left and bottom right corners of the field. By some strange coincidence, that's where Jeeves and wooster take their samples from.) They know that it's composed of 5,000 colored dots, but that's about it. Neither of them has any idea of how the dots are distributed, or anything else besides the fact that it's made up of dots. And of course, neither of them knows that the population is made up of 1,250 red dots (25%), 1,250 blue dots (25%) and 2,500 green dots (50%)  (You can see that this population is extremely well mixed. In fact, there are only two deviations from perfect mixing. They appear in the top left and bottom right corners of the field. By some strange coincidence, that's where Jeeves and wooster take their samples from.) Now Jeeves takes a sample from the top left corner of the population (red line) while Wooster takes a sample from the bottom right corner, (blue line). Each of them then makes a claim about the composition of the population based on their samples.



Jeeves's sample is 50% green, 25% red and 25% blue. So he claims that the population is 50% green, 25% red and 25% blue.
Wooster's sample is 25% green, 25% red and 50% blue. So he claims that the population is 25% green, 25% red and 50% blue.
appreciate
That's quite a big difference. Who's right and why?

The reason Jeeves's argument is better than Wooster's argument is that argument Jeeves's sample is big enough to swallow it's imperfection in the mixing of the population (which means that his sample is representative of the population) while Wooster's sample is so small that it's imperfection crosses the sample border, distorting the result (which means that his sample isn't representative of the population).

In this case, the population was being tested for color. But it could also have been tested for some other property, say size or shape.

Overall, there are three ways in which a generalization can go wrong.

1. The population can be so unevenly mixed that the sample size chosen isn't big enough to get a representative sample.

2. The sampling method can be dependant so that it seeks out or avoids members with the property being tested.

3. The population can have changed since the sample was taken so that it's no longer representative.

Because of this, we tend to be suspicious of arguments from generalization that rely on small samples, old samples, or on non-random samples. When a sample is small (say less than 10%, which is actually a big sample) we would have to be sure that the population is really well mixed. When a sample is old, we would have to have reason to think that the population hadn't changed since it was taken. When a sample is not randomly selected, we'd have to be sure that the sampling method isn't based on a feature of the population that also significantly correlates with the tested property.

Sample Size

We saw above that it's possible to have a sample that's way too small to accurately represent the population it's taken from. However, it is sometimes the case that a population is structured in such a way that even a small sample can be perfectly representative, if it's taken the right way. A population is not always arranged as a chaotic mixture of individuals. Some populations are arranged in such a manner that we can take a very small sample with absolute confidence that the result will perfectly represent the composition of the population. For instance, consider the population of dots shown below. Imagine that we know that the population is structured in the way shown, but we don't know the colors of any of the rows. Now imagine we take the very, very, very, very small sample of exactly four dots comprising the first dot in each of the first four rows, as shown in the top left corner of the image. That's a sample of four out of twenty thousand. That's one per five thousand, which means one fiftieth of one percent, or 0.0002. Is that too small?


Our sample comes out 50 percent red, 25 percent blue and 25 percent green. Given that we know the structure of the population, what are the chances that the population is 50 percent red, 25 percent blue and 25 percent green?

Therefore, the following argument is a red herring fallacy:

It hasn't been proved that the dots in the picture above are 50% red, 25% blue and 25% green because the sample upon which that generalization is based is only 0.0002 of the population, which is waaaaaaaay too small a sample.

Now consider a population of infinitely many dots, part of which is shown below. (The rest of the dots extend off your screen to the right.) This population is structured as you see here, in four rows of dots, each row being composed of dots of exactly the same color.


How big a sample do you need to tell the composition of this population? Will four dots do? But that's an infinitesimal sample!

So this would also be a red herring fallacy:

It hasn't been proved that the dots in the picture above are 25% dark red, 25% blue, 25% green and 25% orange, because the sample upon which that generalization is based is an infinitely small proportion of the population, which has got to be too small a sample. I mean, come on, you can't even put a number on that proportion. It's smaller than the smallest number you can imagine!


Randomness

People sometimes say that all samples have to be taken randomly, or they're no good. This isn't exactly true. There are circumstances where the population structure will make it possible for a small non-random sample to be much more representative than an equally sized random sample. For instance, imagine that someone took a random sample of four dots from the above population? Could we guarantee that each of those randomly chosen dots would come from a different row? Heck no! If we took a random sample of four dots, we would have 24 chances out of 256 of getting the right answer, or less than 10%, whereas, if we take advantage of the population structure in the way shown above, we are guaranteed to get the right answer. So there!

Here is a way this kind of thing might play out in real life. Imagine Cortez works for a company that makes very complicated cables to extremely precise specifications. She works in a department which sells surplus cables to scrap metal companies. Her job is to estimate how much of each cable they plan to sell is copper, how much is steel, how much is aluminum and so on, so the company can set a price. She works from the cable specifications, and has very precise instruments, accurate to one part in one hundred billion with which to check whether the cable they're selling really matches those specs. One day she is told that the company made a hundred million kilograms of a type of cable that nobody wants. And they lost the specifications! She is told to estimate the quanities to the nearest percentage point. The cable is sitting on the docks in Singapore, but they've air expressed a one-kilogram sample of the cable to Cortez for testing. She finds that sample to be exactly 42% copper, 33% steel, 17% aluminum and 8% worthless plastic coating by weight. She then has the following discussion with her boss, Shea.

Cortez: Including our sample, the surplus cable contains 42 million kilos of copper, 33 million kilos of steel and 17 million kilos of aluminum. Those are the stats of our sample, and every metre of that cable has the same cross section as every other metre, so our sample tells us the exact composition of the whole rest of the cable.
Shea: But your sample size is one kilo out of one hundred million, or 0.000,000,001, which is way too small to provide an accurate figure!

Scaefod:

Cortez:     1. The sample is exactly 42% copper, 33% steel, 17% aluminum and 8% worthless plastic coating.
                 2. Every metre of that cable has the same cross section as every other metre.
                 (3. The total weight of the cable is a hundred million kilograms.)                                                                                    
                 C. The surplus cable contains 42 million kilos of copper, 33 million kilos of steel and 17 million kilos of aluminum.        DIRECT

Shea:        1. The sample size is one kilo out of one hundred million, or 0.000,000,001.
                 2. 0.000,000,001 is way too small to provide an accurate figure.                                                                                      
                 (C. We don't know the true composition of the cable because Cortez's figures have not been proven to be reliable.)      COUNTER

Cortez offers a direct argument, because she's saying something about the world.
Shea offeres a counter argument, because he's criticizing Cortez's argument.

Cortez bears the burden of proving that the whole cable has that particular composition. If it turned out that neither of these arguments was any good, it would follow that we just didn't know the composition of this cable.

Cortez   Generalization Argument.                                       Shea  Sample Spoiler Counter Argument.
             Population: A hundred million kilograms of cable              Problem Cited:  Sample too small.
             Sample:   One kilogram of that cable                                Evidence for Problem: Sample is 1 part in 100,000,000
             Age: very recent                                          
             Size:    1/100,000,00                                          

Evaluation: Sample size is not enough to discredit this study. The instruments are so precise Cortez can't be off by more that one part in a hundred billion, which means that her figure for steel could be off by one part in ten million. That's not enough to make even a percentage point either way. Now, if Shea could point to a possible flaw in the manufacturing process, or to possible mistakes in the reports from Singapore, he might have something. But sample size can't discredit the figures in this case.

Fallacy: Shea is committing the fallacy of Red Herring.

Another way to get an accurate result with a very small sample is if a population is perfectly mixed. Imagine another infinitely large population in which individuals are so perfectly mixed that every part of the population looks like the following picture.




Try to find a four-square group, or a contiguous line of four dots that isn't a representative sample for this population.

Now imagine blindly picking dots from random places scattered through the population. How many would you have to pick to guarantee a representative sample? Not many!

Now imagine you work for a petroleum company. You check the composition of oil products so the company can decide how each tanker load will be processed. Your company's tankers contain a pumping system that circulates the oil between all the tanker's oil-carrying compartments. All the oil is moved and turbulance from the pumping process mixes the oil products so thoroughly that every centilitre in that tanker is absolutely identical to every other cubic centilitre in that tanker. Given that a litre is one hundred centilitres, would one litre be a big enough sample to test the composition of the oil mixture in a tanker holding a billion litres of oil products?

The point here is that small sample size may make the sample untrustworthy, but there may be special circumstances that make this particular sample an accurate representative of the population, even though it is way smaller than we would normally accept as a good sample.

If 1% can be an adequate sample, 50% can be inadequate. Imagine that Noah was an educational administrator who had to rely on state grants for his funding. God says get two of every animal, but his superiors insist he spends half of God's money on computers. Thinking outside the box, Noah adapts to the situation by only including one of every animal. What if aliens later came across Noah's Ark bobbing on the flood waters, how big a sample would they need to accurately represent the animal passenger list? Say they picked 50% of the animals at random. Would that give an accurate picture? Would 90% be enough to give a picture that was accurate to within 1%?

How big is big enough? Firstly, the issue of whether a particular sample is big enough doesn't depend on what percentage of the population is included. If the above well-mixed population was only four individuals large, only a 100% sample would be big enough! Anything less than four would leave out at least one color! If the above population was 8 individuals, a 50% sample would do. If it was 16, a 25% sample would work. If it was 32...

The minimum necessary sample size depends on the number of different relevant properties individuals can have, and on the degree of mixing in the population. In the well-mixed population above, the number of different relevant properties is four, because there are four colors, and the population is perfectly mixed. If the number of different properties was larger, or if the population was less well mixed, minimum necessary sample size would be larger.


Old Sample

Imagine you are an atmospheric scientist studying inertium monoxide levels in the atmosphere at various points in history. Like the rest of Earth's air, intertium monoxide does not react with any other gases in Earth's atmosphere. Say that because of the way inertium monoxide is produced and distributed, the level of inertium monoxide in Earth's atmosphere at any point on Earth is never more than ten percent more or less than the global average at that time. (If today's global average is 1%, there's nowhere on Earth where the inertium monoxide levels are lower than 0.9% or higher than 1.1%.) You recover an air sample that's been held absolutely isolated for three thousand years at the base of a really old glacier. (They can actually do this for samples that are several hundred years old.) The sample contains 10% inertium monoxide, so you can conclude that threee thousand years ago, Earth's atmosphere held a global average of between 9 and 11 percent intertium monoxide.

Is the sample too old? Not if the sample was absolutely isolated! Remember, there's nothing in Earth's air that inertium can react with, so the sample can't change over time. Isolation prevents sample gasses from escaping and gasses from later atmospheres from getting in, so it can't change that way either. So, in this case, a three thousand year old sample is enough for a good generalization, provided that all the other factors are taken care of. Notice however that we can't use this sample to generalize about today's atmosphere. Atmospheres can change quite spectacularly over time. Imagine trying to generalize about the air in Los Angeles today based on an air sample taken in 1902! This is why I use the term obsolete, which means that we know that conditions have changed, so that the sample is no good anymore. A sample can be very old without being obsolete, and a sample can be obsolete without being very old at all.

Here's a real-life example. More than about 4 billion years ago, the solar system was nothing but a widely spread out mass of gas and dust particles which was slowly but surely organizing itself into bigger and bigger clumps, many of which banged into each other to make larger clumps. Our Earth was one of those lumps. While the Earth was first forming, it was hot and mostly molten, so the heavier materials gravitated to the center of the lump and the lighter materials were forced up to the surface. The heaviest materials became the Earth's core. Just before the Earth finished forming, a really big lump smashed into it hard enough to kick some of that core material up to the surface on the other side of the Earth. 4 billion years later, scientists found some of that material, figured out what it was, and used it to figure out the exact chemical composition of the Earth's core. Think about it. Not only is the few pounds of material they used a tiny, tiny sample relative to the total size of the Earth's core, that sample is 4 billion years old. However, the Earth's core has been subjected to enormous heat, pressure and mixing by convection, so it's extremely well mixed. Furthermore, we have good reason to think that the composition of that core has not changed in 4 billion years, and that the composition of the pieces of core material that the scientists used hadn't changed either. So in this case, a sample that's about as old as a sample can get on this planet turns out not too old!

Of course, saying

That generalization's no good because the sample's 4,000,000,000 years old!

is a red herring fallacy.

On the other hand, think about trying to do a generalization about computer use, or cell phone use, or home recording equipment, based on data from 1950.

Again, we always need to think about the age of a sample, but we still can't dismiss a generalization based merely on age. If we have good reason to think that the sample hasn't changed since it was taken, and that either the population hasn't changed either, or the generalization is about what the population was like at the time the sample was taken, then the generalization is fine even if the sample is old. Some samples become obsolete very quickly, some stay good for a very long time indeed.


Sampling Method

A generalization can only work if it uses a sampling method that is completely independent of the property being tested. If the sampling method is at all sensitive to that property then it will tend to either seek out or avoid members of that population that have that property. Either way, the result will be skewed.

Some people call this sensitivity "bias." I don't like that terminology. For one thing, "bias" has more than one meaning, and not all it's meanings have anything to do with the accuracy of generalizations. And a sampling method can be very biased while still giving a very accurate result. The thing to remember about "bias," is that it only counts if the bias is relevant to the property being tested. If you're sample is biased, but you can show that the bias has nothing to do with the property being tested, then that bias gives you no reason to throw out the study.

Let's go back to assessing the tipsiness of Hogmanaying Scots. Say we happen to know the names and addresses of three significant groups of Scots. We know the names and addresses of all Scottish accountants, all Scottish teetotalers, and all Scots who have been convicted of drunk driving at least three times. Say we examine every member of each group to see whether he or she got tipsy last Hogmanay. And say we got the following results.

1. 67% of all Scottish accountants got tipsy last Hogmanay.

2. 0% of all Scottish teetotalers got tipsy last Hogmanay.

3. 99% of all Scots who have been convicted of drunk driving at least three times got tipsy last Hogmanay.

These results can't all be right. At best, only one is right. So which of these figures is more reliable? The answer is, whichever one is least sensitive to the property being tested. What do accountants have to do with tipsiness? Nothing that I can think of! But teetotalers are people who habitually abstain from alcoholic beverages. (Strange, but true.) So of course none of them got tipsy on Hogmanay. Are all Scots teetotalers? I don't think so! So that sample is definitely dependent on the property being tested. On the other hand, habitual drunk drivers can be expected to drink more than regular Scots, so that sample is dependant too. (Notice that one of them is negatively dependent, in that it avoids the property being tested, and the other is positively dependent, in that it seeks out the property being tested.) So, since we can't find an obvious link between accountancy and tipsiness, sample number one is the only independant sample.

It can be very difficult to tell whether or not a sampling method is dependant. Consider the following simple population. It consists of small. medium and large triangles, circles and squares in the colors yellow, magenta and cyan.


Suppose you couldn't see the whole population, but could sample it in some organized way. Suppose also you were asked to estimate the distribution of colors among the shapes, based on a smaller sample. And let's say your first attempt uses a sampling method that lets you pick out all the large shapes, ignoring the small and medium sized individuals. What kind of sample will you get? Let's see. 

Will this sample give you an accurate picture of the overall distribution of colors in the population? In this population the feature of having large size is perfectly correlated with the feature of being yellow, so picking out the large individuals results in only picking out yellow individuals. So if a population has a structure such that one possible result (yellow color) is more strongly correlated with another feature (in this case large size) then you can't use that feature to pick out your sample. But watch if we pick out just the triangles.

This gives a better result because, in this population, shape is not correlated with color in any way.

The trick to telling whether or not a sample id dependant is to look at the way the sample was obtained. If it was obtained in a way that has nothing to do with any of the possible outcomes of the study, then it is not dependent. If, however, the method by which the sample was chosen is logically connected to the properties the sample is supposed to test for, then that's a dependancy, and the argument is no good. Consider the following sampling methods.

1. Testing American reactions to the war in Iraq by mailing questionnaires to the membership of the American Pacifists Association.

2. Testing the distribution of blood types across the United States by taking blood samples from members of the Mayflower Society, a group which restricts its membership to people who have at least one ancestor that came over to America on the Mayflower.

3. Assessing the bodily proportions of 18th-century Americans by measuring antique clothes preserved by historical societies.

Obviously, the first sampling method is no good because we would be taking our sample from a group that is already self-selected to be against any war. The second sampling method is also dependant because the Mayflower passengers came from a very small region in Europe whereas the vast majority of other immigrants to the United States came from other regions, and continents, and blood type is very highly correlated with ancestry. Finally, there is the fact that until recently, good quality clothing (the kind that is likely to be preserved) tended to be reused as long as it could be made to fit new people. Larger clothing was easier to alter than smaller sized clothing, so it tended to be reused until it wore out. Smaller sized clothes tended to be put away in the hope that someone would come along who could use them, so smaller sized clothing is much more likely to have been preserved than larger sized clothing. Therefore, the third sampling method is also dependant.

Again, pointing out an irrelevant "bias" is just a red herring. Only real dependancies count.

Counter-example.

Later in the course we will go more deeply into the issue of what I call "explanation" arguments. (The way they work is easy to understand but very difficult to explain, so I save it for later.) However, there is one specific type of explanation argument that is very relevant here. This is the argument by counter-example, and it can be a very effective argument against general statements. For instance, if someone was to make the statement that all conservative politicians are draft-dodging hypocrites, we could offer the counter-example of John McCain, a conservative politician who not only fought in the Vietnam war, but also spent years in a Vietnamese prison camp. The only way anyone could explain the existence of John McCain would be to admit that not all conservative politicians are draft-dodging hypocrites. Thus, we have an argument to the effect that not all conservative politicians are draft-dodging hypocrites.

Here is a dispute involving an explanation argument, followed by a scaefod of that dispute.

Adonis. Much as I would like to believe that all cops are honest, I keep coming back to the numerous, well-documented cases in which police officers lied on the witness stand, planted evidence or otherwise perverted the justice system. So I just have to admit the uncomfortable fact that not all cops are honest.
Greta. You don't know what you're talking about. I've watched every episode of Cops ever made, and none of the real police officers featured in those shows ever did anything remotely dishonest, so it has to be true that all cops are completely honest.

Adonis    1. There are many well-documented cases of dishonest police officers.
                  C. Not all police officers are completely honest.

Greta        1. All of the real police officers featured on the show Cops were honest in the footage shown.
                  C. All police officers are completely honest.

Greta bears the burden of proof here because she is making a broad generalization that goes beyond what we would expect, given what we know about human groups and institutions. History shows that it is very unusual for any group of people to be such that it contains no dishonest members. If Greta wants us to believe that all cops are honest, she is going to have to give a strong argument to that effect. In the absence of such an argument, the null hypothesis would be that police officers are like any other human group, and have some honest and some dishonest members. This is Adonis's conclusion, so he definitely does not bear the burden of proof.

Adonis makes a direct argument because he is supporting a claim about the world, not about Greta's argument.
Greta makes a direct argument, because she is supporting a claim about the world that contradicts Adonis's conclusion without saying anything about Adonis's argument.

Evaluation.
Adonis offers examples of cops who are less than completely honest, which fact cannot be explained unless we assume that not all cops are honest, so Adonis has a very strong argument. Greta offers a generalization based on her experience watching the show Cops. These are real police officers, and the footage is pretty recent, so there's no problem there. Sample size is okay, since a few hundred individuals is usually big enough to make generalizations about a very large population. However, Greta's sampling method is fatally dependant. Dishonest cops can easily refrain from doing dishonest things on camera, so this sample doesn't even prove that the police officers who appear on the show are honest, let alone those officers who choose not to be filmed. Furthermore, even if all the officers who appeared on the show were completely honest, we could still easily explain that fact while also assuming that Adonis's conclusion is true. Adonis's evidence provides counter examples to Greta's conclusion, so we have reason to think that Greta's conclusion isn't true. Since Adonis doesn't say that all cops are dishonest, so Greta's evidence is not a counter example to Adonis's conclusion. This means that if these two arguments were the only arguments we had, we would have to conclude that Adonis is right that not all police officers are honest.

Notice that counter-example can be a very powerful argument. A good counter example pretty much always defeats any generalization that purports to cover all of a population. A single counter example won't, of course, defeat a generalization that only covers 90 percent of a population. To defeat that kind of generalization you would have to show that more than 10 percent of the population lack the property in question. And so on.


Being small or old doesn't necessarily always make a generalization bad. Sometimes, we will have reason to believe that a small sample, an old sample, or a small, old sample really is representative enough to support a generalization. And a sampling method may be structured in a way that looks weird without it being dependant. This means that there are a total of four fallacies we can associate with generalization arguments.

Fallacies

There are a couple of fallacy names that would be helpful to know before we talk about how to evaluate generalization arguments. They are hasty generalization and red herring. "Hasty generalization" is the name for any generalization where the sample logically fails to support the conclusion. "Red herring" is the name for any argument where a crucial premise is logically irrelevant to the truth or falsity of the conclusion.

Hasty Generalization

The term "hasty generalization" is really too vague to be useful, so I break it down into four seperate fallacies: Inadequate sample, Obsolete sample, Dependant Sample and Anecdotal Evidence. The key to determining whether an argument commits hasty generalization is to ask whether the available facts allow a reasonable alternative explanation for the state of the sample. Here are the generalization fallacies in more detail.

1. Inadequate sample. The population clearly has not been shown to be so evenly mixed that a sample of this size can be reasonably assumed to properly represent the population. Remember that 1% can sometimes be big enough while 90% (or more) can sometimes be too small.)

The International University on the Moon has over 20,000 students from all of Earth's 140 or so countries. I've taken an absolutely random sample of 10 students out of those 20,000, and 5 of those students were from Armenia, so we know that 50% of the students on the Moon are from Armenia.

Imagine that 143 countries are represented on the moon. In that case, a ten-student sample will miss at least 133 of those countries. This means that a sample needs to be at least 143 students to have a hope of being adequate, and we would probably want about 300 to have anything like a reasonable sample.

2. Obsolete sample. The population clearly has not been shown or clearly cannot be assumed to be unchanged since the sample was taken, so it's clearly possible that the population has changed, making the generalization out of date. (Remember that 15 billion years isn't necessarily too old while an hour isn't necessarily recent enough.)

In 1843, 35% of all American families owned at least one buggy whip, that means that there's a 35% chance that there's a buggy whip in your house.

Considering that Americans almost completely stopped driving horse-drawn buggies once automobiles became widely available, information from when buggies were widely used is not going to represent present transportation related realities.

3. Dependant sample. The sampling method clearly has not been shown or clearly cannot be assumed to be random with respect to the property being tested, so that it's clearly possible that the sample fails to accurately represent the population. (Remember that a "bias" that is not relevant to the property being tested cannot be a problem.)

Did you know that they recently held a school assembly where they publically interviewed 20 randomly chosen graduates of the schools Substance Control and Abuse Rejection Enterprise program, and 100% of those SCARE graduates reported that they've never tried drugs!

Considering that drugs are illegal, and that a student who publically admits to having tried drugs is going to be in a lot of trouble, it wouldn't be surprising if some or all of those students were lying.

4. Anecdotal Evidence. Here the arguer fixes on a particular story and tries to use it to support a generalization. The problem is that the anecdote could easily have been picked precisely because it supports the point the arguer wants to make, and might be screamingly atypical of the population he wants to generalize about.

Handgun Control, Inc. faked statistics on gun violence. That proves all gun-control activists are liars.

That Mensa member tried to murder the people next door with thallium, and wrote snotty articles about it in the Mensa newsletter. That proves that all smart people are evil.

Of course America was as deeply involved in witch burning as europe was. Didn't you hear about the Salem Witch Trials?

Keagan. Okay, I'll admit that some cops are racist. But you'll have to give me some pretty convincing evidence before I'll believe that all cops are racist.
Aylin. But didn't you see the Rodney King videotape? That videotape showed five white LAPD officers repeatedly beating a prone, unresisting black motorist. They just kept whaling on him, hitting him over and over again. It was a savage, stupid beating that King would not have gotten if he had been white. That proves all cops are racist thugs.

The key here is that the arguer gives a very salient example. That is, he gives an example that sticks out, or otherwise makes a deep impression on the listener. But salience isn't significance, and an example can be very salient without being at all representative.


Red Herring

Apart from the fact that Red Herring is a very common fallacy, I mention it here because people often attack generalization arguments on the basis of sample age, sample size or sampling bias when these issues are completely irrelevant to the strength of the argument. Therefore, an arguer commits red herring if:

1. His criticism of a generalization is based on sample age when we have no reason to think that either the population or the sample has changed since the sample was taken.

2. His criticism of a generalization is based on sample size when we have no reason to think that this particular sample is too small for this particular population.

3. His criticism of a generalization is based on a bias in the sampling method when we have no reason to think that this particular bias has anything to do with the property we're testing for.



Appropriate and Inappropriate Criticism

Someone who wants to oppose a generalization argument must do more than mention things that might be wrong with the study. He has got to show that at least one one of the things that might have gone wrong with any study actually has gone wrong with this particular study. Consider the following argument pair and pick out the "opposition" that most accurately characterizes the opposing argument.

Marsala. I've seen a summary of the results of all the studies that have ever been done comparing the murder rates in states with and without the death penalty. After all the corrections have been made to account for all the variables and other demographic factors, it turns out that there's no evidence that the death penalty has any deterrent effect whatsoever. Since the effect would show up in a difference in murder rates if it existed, we should conclude that there is no deterrent effect.
Naan. You're misunderstanding the nature of statistics. Actually, the evidence for the deterrent effect of the death penalty holds up pretty well. You have to understand that there are all kinds of things that can go wrong with a study. There may be bias, for instance. Or the data may be incomplete. Or the researchers might have compared demographically dissimilar groups. So it should be clear to you now that we can discount these studies, and should affirm that the death penalty does have a deterrent effect.

1. "Naan argues that Marsala is wrong about the implications of the studies he, Marsala, cites in support of his claim that the death penalty has no deterrent effect. Naan argues that it does not matter that the studies compared murder rates with and without the death penalty, and make corrections for all the variables and other demographic factors. Naan gives reasons why we should discount these studies that show no evidence for a deterrent effect, and reject Marsala's conclusion that the death penalty has no deterrent effect." (Answer)

2. "As Naan says, Marsala misunderstands the nature of statistics, and evidence for the deterrent effect of the death penalty holds up under scrutiny. These claims are supported by the following facts. Studies can go wrong because of bias. Studies can fail because the data is incomplete. And studies can fail because the researchers compared demographically dissimilar groups. From these facts, we should conclude that these studies should all be discounted, and that the death penalty does in fact have a demonstrable deterrent effect." (Answer)

3. "Naan makes a series of serious errors in reasoning. First, she falsely claims that Marsala misunderstands the nature of statistics. Second, she makes the completely unsupported claim that the evidence for the deterrent effect of the death penalty holds up under scrutiny. Finally, she lists three things that can go wrong with studies, (bias, incomplete data, comparison of demographically dissimilar groups), without giving us any reason to believe that any of these things apply to the studies she's criticizing. Finally, she illegitimately infers that these general facts that apply to all studies support the claim that we should discount these particular studies and, again without foundation, goes on to claim that we should affirm that the death penalty does have a deterrent effect." (Answer)

4. "Naan makes two claims. The first is that Marsala misunderstands the nature of statistics. The second is that the evidence for the deterrent effect of the death penalty holds up to scrutiny. She supports these claims by pointing out that studies in general can go wrong because of bias, incomplete data or the comparison of demographically dissimilar groups. From this, Naan concludes that we can discount the studies that Marsala cites in support of the conclusion that the death penalty has no deterrent effect." (Answer)

Now think about it. Has Naan proved that any of those things went wrong with any of the studies under consideration here? If he hasn't given a specific reason to think that there's a specific problem with a particular study, then he hasn't given a reason to discount that study. Reference to things that can go wrong with studies is a red herring unless a secific problem can be shown to exist with a specific study.

Here are some argument pairs to work on. Critique whatever arguments you can find. Identify all potential problems with each argument. (maybe inadequate sample, maybe dependant sample, maybe obsolete sample, maybe anecdotal .... ) and identify any reasons why they might not be real problems. (Not too small because..., This "bias" not relevant because..., Not too old because.....) Give your overall assessment of each argument. Don't forget to also look out for problems that don't fit easily into the categories I have described.

A. Keyshawn. My survey says that Americans don't particularly care about the proposal of adding a small federal tax on all computers and modems sold in the United States. My people visited over a thousand grocery stores and speciality markets in all fifty states. They selected people from all walks of life and all income levels. They asked twenty thousand people what they thought about the proposed tax. Most hadn't heard of it, and didn't care about it when they did hear. Those who cared were evenly distributed between mildly for and mildly against.
Dominique. Well, your information is wildly wrong, probably because of the small sample size. My company found a way to reach one hundred thousand people in a very short period of time. We did an e-mail poll of names selected at random from a very large database. Ninety-five percent of our respondents had heard of the tax, and eighty percent of all respondents were strongly against it. (Answer)
 
B. Deion. A lot of Muslims live in my neighborhood. There's a mosque just up the street from my house. So I meet many Muslims from all different countries as neighbors and friends. None of them want to forcibly convert anyone to Islam. None of them know anyone who wants to forcibly convert people to Islam. In fact, no Muslim I know knows of anyone who wants to forcibly convert people to Islam. So I don't think more than a tiny minority of muslims want to forcibly convert people to Islam.
Aryanna. You're so naive. Haven't you heard of the muslim wars of conquest? Every Muslim in existence then was committed to forcibly converting everyone in the world to Islam. And they acted on this commitment, marching in massive armies into Arabia, North Africa, Europe and Asia. They converted everyone they met at sword point, and killed everyone who wouldn't convert. These actions were universally applauded in the Muslim world, so a majority of Muslims strongly support the conversion of people to Islam by force. (Answer)
 
C. Deangelo. I wonder how many people believe in Bigfoot nowadays. The story of a big, hairy "wild man" living in the remote forest areas of America used to be very popular. But since the guy who started the story has come forward with the big wooden foot that he used to make the original footprints, I expect that most people nowadays don't really believe that Bigfoot is real.
Micah. Well, your information is really out of date. Just a few weeks ago, the network of Fake-Jamaican Psychics gave a telephone survey to everyone who called in for psychic or astrological advice. They had 27 million callers, and 74 percent of those 27 million asserted that they firmly believed in the reality of Bigfoot. Two weeks ago is very recent. 27 million is an enormous sample for this kind of poll. No other opinion poll has used a sample size of more than about 10,000, and many of those polls are considered extremely reliable! So we can take it as proved that about 74 percent of Americans believe in Bigfoot. (Answer)
 
D. Pierre. The latest AARP survey says that American seniors are living longer and healthier lives than ever before. Old people make up around 10 percent of American society, and respondents to the AARP survey turned out to be both significantly healthier, and to have on average lived considerably longer than a demographically identical group surveyed only five years previously. I think this survey is reliable, because it is based on responses from nearly the entire membership of the AARP, which is of course composed entirely of seniors, and was supervised by the best statistical survey analysts available.
Sonya. You're forgetting one thing. The AARP only makes up just over 5 percent of the American population. How can you make any kind of serious generalization based on a sample that is just 5 percent of the population? (Answer)
 
E. Freddie. I've got to say that in a weird way my respect for conservatives has increased during the present crisis. I've talked to a lot of conservatives about the present situation and most of them present very reasonable case for their own side. They are not a bunch of bloodthirsty warmongers, or knee-jerk jingoists who support any military action no matter how ill-advised. Rather, the overwhelming majority of the ones I've talked to are extremely upset by what they see as the necessity for military action, and although I firmly disagree with their reasoning, I have to say that most of them have taken a great deal of time and effort to think through the issues. Let's face it, there's plenty of intelligent conservatives out there.
Martina. I don't know how you can say that there are plenty of intelligent conservatives out there. I've listened to A.M. radio dozens of times and every Conservative talk show host I've ever heard has been an ignorant, irrational blowhard who does nothing but disparage liberals without ever bothering to find out what any actual liberals are actually saying about anything! Yes, there's a lot of variety in these talk show hosts. There are loud blustery idiots, and quiet vicious idiots, and pedantic boring idiots, and self-important patronizing idiots. But there's nobody who's willing to even begin to talk about the real issues and arguments! (Answer)
 
F. Gino. I'm worried about the sulfur content in that load of crude oil you've got tied up at the docks there. I've just heard that it has come from an oilfield where the crude usually has a high sulfur content. That's a large capacity supertanker you've got there with over a hundred separate storage tanks, so if I load all your oil into my refinery, I could end up contaminating my entire works with sulfur products.
Elsa. I anticipated your concern, and I dipped out this five gallon sample from the No. 42 hold before I came over to your office. Your own lab has certified that it has a very low sulfur content, so you don't have to be concerned about the sulfur content of my oil. (Answer)
 
Just to make things difficult, the following exercises have hints instead of complete answers.

H. Kathy. My cousin just came back from a business trip to Viet Nam. She said the people were nice enough, but she thought there was an undercurrent of resentment and suspicion towards Americans among most of the people she dealt with there. I guess the Vietnamese over there are still not quite as friendly towards American business people as people in other parts of the world.
Madisen. Your cousin is dead wrong. All the people in Viet Nam love and admire Americans. After the Japanese occupiers surrendered, Ho Chi Minh and other Vietnamese leaders welcomed the Americans in as liberators and supporters of Vietnamese independence. Why, love and admiration for America was part of the Vietnamese language at that time. People used to say "oh, to be as rich and wise as an American!" Does that sound to you like people who are suspicious and resentful of Americans? (Hint)
 
I. Gideon. You know you thought that we would never be able to get any kind of accurate idea about the composition of the Earth's core? Well, scientists have discovered that a massive meteor or asteroid whacked into the Earth while it was still relatively hot, and the shock wave kicked up some of the core material through the soft mantle and crust. The crust was solid enough by then to hold this material in place, and they were able to recover a sample. The samples were 90% iron and 10% nickel, so the Earth's core is 90% iron and 10% nickel
Anaya. Wait a minute! That asteroid impact must have been over ten billion years ago. Ten billion years must be the oldest sample ever taken in science! We commonly discard hundred-year-old samples as too old, and we don't even look at some thousand year old samples. Your sample is ten million times as old as that, so it can't possibly be any good. (Hint)

J. Marsala. I've seen a summary of the results of all the studies that have ever been done comparing the murder rates in states with and without the death penalty. After all the corrections have been made to account for all the variables and other demographic factors, it turns out that there's no evidence that the death penalty has any deterrent effect whatsoever. Since the effect would show up in a difference in murder rates if it existed, we should conclude that there is no deterrent effect.
Naan. You're misunderstanding the nature of statistics. Actually, the evidence for the deterrent effect of the death penalty holds up pretty well. You have to understand that there are all kinds of things that can go wrong with a study. There may be bias, for instance. Or the data may be incomplete. Or the researchers might have compared demographically dissimilar groups. So it should be clear to you now that we can discount these studies, and should affirm that the death penalty does have a deterrent effect. (Hint)

K . Emerson. I'm worried about all the money the government is spending on the DARE program. It takes time away from education, and no one has ever presented evidence that it does any good. That time and money could be spent on academic subjects, or even art and music programs, which have been proven to have a positive effect, so I think we should abolish DARE.
Joana. Well, I've got the evidence you need! Back in the '70s, before the government started DARE and other anti-drug programs, 20 percent of high school seniors said that they had tried drugs. Last year, they administered the same survey to a group of high school seniors who had been through the DARE program, and only 10 percent of them said that they had tried drugs. (Hint)

This next one I plan to discuss in class, as well as the homework exercises.:

L. Alessandro. I was just reading a report by some feminist group or other. They took World Health Organization and United Nations statistics for the amount of the world's work that is done by women and compared it to the amount of the world's wealth that is actually controlled by women. It turns out that two-thirds of the world's work is actually done by women while only five percent of the world's wealth is controlled by women.
Liliana. That is complete and utter nonsense! Don't you know that PARADE magazine reported that 86 percent of all the personal wealth in the United States is owned by women! 86 percent! Now do you see that those feminists don't know what they're talking about?

Logic Homework. (For my logic and critical thinking class.) Circle the name of the (nearest to) right population, sample and property for each of the following arguments.You can do this exercise on your own lined paper, (if it doesn't have curly edges from ripping it out of a spiral notebook), or you can use the Homework Answer Sheet 1. Taya. I know you think that Mount Stoney is all granite, but I've come across a set of rock samples, which I'm calling "The Stoney Rocks," that were taken in 1851 by competent geologists who took samples of rocks from various places inside and on the surface of Mount Stoney. It was a perfectly conducted study, and the samples are all andesite, not granite! So Mount Stoney isn't all granite. Population:   The Stoney Rocks     Everything granite     All the rock in Mount Stoney    Everything Andesite   Geologists Sample:         The Stoney Rocks     Everything granite     All the rock in Mount Stoney    Everything Andesite   Geologists Property:       Made of granite?   Competent geologist?   Sampled by Taya?     From Mt Stoney?    One of the Stoney Rocks? (Note: The population doesn't have to be a bunch of individuals. It could be a massive, monolithic thing that could be divided up into parts, like an ocean, a planet, a desert, or a really big piece of cheese.) 2. Martina. I don't know how you can say that there are plenty of intelligent conservatives out there! I've listened to A.M. radio dozens of times and every Conservative talk show host I've ever heard has been an ignorant, irrational blowhard who does nothing but disparage liberals without ever bothering to find out what any actual liberals are actually saying about anything! Yes, there's a lot of variety in these talk show hosts. There are loud blustery idiots, and quiet vicious idiots, and pedantic boring idiots, and self-important patronizing idiots. But there's nobody who's willing to even begin to talk about the real issues and arguments! Population:    Conservatives      Intelligent conservatives     Conservative A.M. radio hosts      Irrational blowhards          Sample:         Conservatives      Intelligent conservatives     Conservative A.M. radio hosts      Irrational blowhards Property:       Conservative?      Intelligent?     A.M. radio host?      Liberal?      Naked? 3. Maribel. Few, if any Americans are carriers for the Neiman-Marcus gene. I tested every attendee at the O'Shaughnessy family reunion, and not one of them had the Neiman-Marcus gene. This is a good survey because it included over 100,000 people taken from locations widely distributed over all 50 states. Population:      All Americans     All carriers of the Neiman-Marcus gene.      O'Shaughnessy reuinion attendees Sample:           All Americans     All carriers of the Neiman-Marcus gene.      O'Shaughnessy reuinion attendees Property:      Are they Americans?    Do they carry the Neiman-Marcus gene?    Were they at the O'Shaughnessy reuinion? 4. Deion. A lot of Muslims live in my neighborhood. There's a mosque just up the street from my house. So I meet many Muslims from all different countries as neighbors and friends. None of them want to forcibly convert anyone to Islam. None of them know anyone who wants to forcibly convert people to Islam. In fact, no Muslim I know knows of anyone who wants to forcibly convert people to Islam. So I don't think more than a tiny minority of muslims want to forcibly convert people to Islam. Population:      All Muslims      Muslims Deion knows      Muslims who want to forcibly convert people.       Sample:           All Muslims      Muslims Deion knows      Muslims who want to forcibly convert people.       Property:        Are they Muslims?      Are they Muslims Deion knows?      Do they want to forcibly convert people?

Writing Homework (For my critical thinking and writing and critical thinking and discourse classes.) Pick three dialogs from the exercises that preceed the homework and, for each one, write a paragraph explaining exactly what is wrong with the weaker of the two arguments.It is important that you work on three dialogs for this homework.

Possible Quiz Questions (This ain't homework! Memorize the answers for next class, cuz there will be a quiz.)
i. What's the difference between hasty generalization and red herring?
ii. Explain the three ways to commit red herring when criticizing a generalization.
viii. What are the three ways a generalization argument can go wrong?
iii. Is 1% always too small? Is 50% always big enough? Explain
iv. Is 1000 years always too old? Is 1 year always recent enough? Explain.
v. What kinds of biases are a problem? What kinds of biases are not a problem?
vi. What does it mean to say that a sampling method is "dependant?

Copyright © 2006 by Martin C. Young

Next page.

---

This Site is Proudly Hosted By.