Science Test or IQ Test?

I've been going through old regents exams and pulling out images to make flashcards (which I will post when they are finished). I've got about 500 100 images now, but some of them are more or less duplicates that I have to weed out. Going through the multitude of different diagrams and models is getting me a little frustrated about the state of science education and education in general.

One of Willingham's chapters deals with the process of transfer of knowledge. Using knowledge of how Scenario A works can help us figure out how an analogous Scenario B works - if we recognize the two scenarios as analogous, which isn't always so easy. It requires a deep understanding of the underling structures of Scenario A. I'll illustrate the point with the following two images from old regents exams.
This is an energy pyramid.* It shows a producer level (A) followed by primary (B), secondary (C), and tertiary (D) consumers. There are several statements we can make about the pyramid. A might represent plants, which are eaten by B, herbivores, which can be eaten by C, omnivores or carnivores, which can in turn be eaten by D, other carnivores. We can say that D depends on C, which depends on B, which depends on A (which depends on the sun and inorganic substances from the environment, not represented in the diagram). Using basic logic, then, we can say that D and C depend indirectly on A. We could also say that D depends indirectly on the sun and so on.

Finally (for now) we can say that there will always be more energy available at the A level than the B level in a stable ecosystem. More energy at the B level than the C level. More energy at the C level than the D level. The reason for this phenomenon is that organisms do not store all the (food) energy they consume from the level below them. Instead, they use it for their survival needs - staying alive burns energy, so that every step up the pyramid, the energy consumed is lost as heat (when animals use energy, it is transformed into low-grade, useless heat that is lost to the environment).

If your an educated adult reading this and your head is spinning already, imagine what this must do to the the kids. We do spend some time teaching these concepts and using more concrete examples - so we might use grasses at level A, then a grasshopper to represent level B, a frog for level C, and a snake to represent D. Still, for most kids this is a lot of abstraction and they are likely to come away with relatively shallow understanding (they haven't had physics or chemistry and heat is not so easy to grasp), but if they see the energy pyramid on an exam they will at least have some idea what concepts to key into in finding an answer to whatever question might be posed about it.

Now consider the following image used to test understanding of the concept just described in an energy pyramid. What do the squiggly arrows represent? Now, I just set you up for the answer by providing you with the analogous situation, but the kids taking the exam have no such clues. Unless they were directly taught this visual model, they must somehow connect the seaweed as Level A, the small fish as level B, and so on, then remember the point of that pyramid was the loss of energy at each level, so the arrows must represent energy lost to the environment as heat.
That's a lot to expect. I remember grading the exam (June '09) with this question and pulling my hair out at how many kids got it wrong. Some of the kids after the exam told me they thought it was sperm, but sperm wasn't one of the choices! They just weren't able to transfer the knowledge. I have since incorporated this kind of model in my instruction, but really, a lot of the success on the exam depends on kids' basic reasoning skills and not necessarily knowledge of biology. A little knowledge goes a long way if you are capable of abstraction and transfer, not skills that many 9th & 10th graders possess in particular abundance.

Stronger students could have figured out the answer by eliminating the other choices, but slower students are easily overwhelmed by the language and logic necessary to use that strategy effectively. To invoke Willingham's analogy again, imagine if your first driving experience were to involve navigating through midtown Manhattan in fast moving traffic. Although the signs and signals and dangers and other cars are right in front of you, plain to see, you would have a hard time processing it all because so much information is coming at you at once. Some kids look at a regents exam and their eyes glaze over, all the words start to run together and nothing makes sense. Whole pages of easy questions near the end are left blank sometimes, the equivalent of stopping in the middle of the street, getting out of the car, and just walking away.

Ultimately, one of the biggest challenges we face is getting students to a level of deeper understanding so that they can analyze and answer questions about a particular concept that use different visual models or different language than they have been taught directly. This is no small task given the number of concepts we have to cover, the limited time (6 periods per week including lab time), and the deficiencies in both background science knowledge and general high school-level vocabulary & reading skills. There's much work to be done!


*It could also be a pyramid of biomass, which can ultimately be thought of as "energy." but the regents exam almost always presents it as a pyramid of energy.