Be Careful While Reading Research Results
Caveat emptorLet the buyer beware. The reader should keep in mind that this applies to receiving and consuming products of research as well as buying a piece of land or a used car.
Context
It appears good on the face of it. Researchers Pennings, Seel, Van Pelt, Sikkink, and Wiens explain in their report that Christian schools . . . have served a vital role in the educational landscape of North America for over 400 years . . . and that from about 1600 . . . until late in the 18th century, the purpose of education in the U.S. was centered on religion . . . (p. 9). These authors also rightly note the following: . . . All schools are religious in nature, and therefore parents desire schools in which congruence can exist for their children between home, religious institution, and school (p. 11).1
The researchers then explain why they are undertaking this project: The Cardus Education Survey has just this purposeto determine the alignment between the motivations and outcomes of Christian education, setting a benchmark for further study of Christian schooling. The reader quickly finds out that Christian education, in this study, includes homeschooling by Christian parents but is not focused on home-based education.
Their survey of graduates (i.e., from secondary school, high school). . . which included participants from Catholic, Protestant, non-religious private, public, and homeschool graduatesfocused on educational and occupational attainment, civic and political engagement, spiritual formation, marriage and family as well as social psychological outcomes in the young adult years (p. 44).
Findings
So . . . what did they find? They found some things that might negatively surprise advocates of homeschooling. Home-educated graduates are the most likely to get divorced? Most likely to feel helpless in dealing with the problems of life? Most likely to lack any clear goals or sense of direction? Most likely to feel prepared for a vibrant religious and spiritual life?
Homeschool graduates are the least likely to be involved in political campaigns? Least likely to spend much time volunteering or going on mission trips? Protestant Christian school and homeschool graduates hold more strongly to the belief that morality is unchanging and absolute?
But before your columnist provides any more of these researchers findings, clouds the research waters, or is quoted out of context, several important points must be made. First, researchers must operationally define important terms in their studies. In this study, homeschooled was defined as a participant reporting he was homeschooled in high school. That is, the researchers only reported that the adults were homeschooled, Catholic schooled, public schooled, and so forth for some of their high school years; no information was given on whether they attended that type of schooling for two, eight, or twelve years. Further, the home educated were categorized as either religious or nonreligious, simply depending on whether their mothers attended religious services regularly. That is all that is known about these adults educational background.
Second, about two thousand randomly selected Americans, aged 23 to 49, were sampled and studied. This sample appears to have been planned and executed well. However, of those, fewer than ninety were home educated at all. And only some of these fewer-than-ninety had a religious homeschool mother during high school. This is a small sample size from which very few dependable conclusions can be drawn.
Finally, this studys findings directly contradict the findings of several other studies about adults who were home educated. For example, Ray found adults who were home educated to be more engaged in direct civic involvement than the general population of the United States. They were more involved in activities such as working for a candidate/political party/political cause, voting in national/state elections, and participating in a protest or boycott.2 Contrariwise, Pennings et al. state that adults from religious homeschooling were noticeably less civically involved on several measures than adults from several other categories of schooling.
Conclusions
Pennings and his colleagues have developed a valuable research concept. They are trying to understand to what degree the outcomes of Christian education match the motivations of adults who design and conduct the Christian education. It should be clearly pointed out, however, that this study has not . . . allowed for the establishment of quality benchmarks . . . (p. 36) regarding the thinking, religious beliefs, and actual behaviors of adults who were home educated. They might have quality benchmarks regarding those who attended state (public), Roman Catholic, and Christian schools but not for those who were homeschooled by Christian parents.
One might wonder: Do reports like this matter in the world in which we live? For better or worse, yes. Research is supposed to get at truth and accurate representations of reality. God says the following: By wisdom a house is built, and by understanding it is established; by knowledge the rooms are filled with all precious and pleasant riches. A wise man is full of strength, and a man of knowledge enhances his might, for by wise guidance you can wage your war, and in abundance of counselors there is victory.3 One can get wiser by reading and heeding true things but not from integrating skewed or too-limited representations of reality into his thinking.
Unfortunately, research like this often shows up in widely read publications. For example, the magazine Christianity Today reported on the findings of Pennings et al. and presented the culturally perceived positive and negative findings about the religiously homeschooled as if the data were reliable and properly comparable to data from other adults.
The authors of this study should have been very careful to explain the notable limitations of their study regarding the home educated. Their caveats should have been clear enough that others reporting on their study would find it difficult to misrepresent reality about any of the statistics in the study.
Caveat emptorLet the buyer bewarewhen it comes to reading and using research reports. Remember, the buyer or digester of researchers findings and conclusions must remember the . . . warning that . . . the goods he or she is buying are as is, or subject to all defects.4
Authors Note: Please feel free to send your questions about research related to home-based education and raising children to [email protected]
Endnotes:
1. Pennings, Ray; Seel, John; Van Pelt, Deani A. Neven; Sikkink, David; & Wiens, Kathryn L. (2011). Cardus Education Survey: Do the Motivations for Private Religious Catholic and Protestant Schooling in North America Align with Graduate Outcomes? Hamilton, Ontario, Canada: Cardus. www.cardus.ca, www.wnd.com/files/2011/07/110826cardus.pdf
2. Ray, Brian D. (2004). Home Educated and Now Adults: Their Community and Civic Involvement, Views About Homeschooling, and Other Traits. Salem, OR: National Home Education Research Institute, www.nheri.org.
3. Proverbs 24:3-6, ESV.
4. Retrieved December 12, 2011, from legal-dictionary.thefreedictionary.com/Buyer+beware.
Context
It appears good on the face of it. Researchers Pennings, Seel, Van Pelt, Sikkink, and Wiens explain in their report that Christian schools . . . have served a vital role in the educational landscape of North America for over 400 years . . . and that from about 1600 . . . until late in the 18th century, the purpose of education in the U.S. was centered on religion . . . (p. 9). These authors also rightly note the following: . . . All schools are religious in nature, and therefore parents desire schools in which congruence can exist for their children between home, religious institution, and school (p. 11).1
The researchers then explain why they are undertaking this project: The Cardus Education Survey has just this purposeto determine the alignment between the motivations and outcomes of Christian education, setting a benchmark for further study of Christian schooling. The reader quickly finds out that Christian education, in this study, includes homeschooling by Christian parents but is not focused on home-based education.
Their survey of graduates (i.e., from secondary school, high school). . . which included participants from Catholic, Protestant, non-religious private, public, and homeschool graduatesfocused on educational and occupational attainment, civic and political engagement, spiritual formation, marriage and family as well as social psychological outcomes in the young adult years (p. 44).
Findings
So . . . what did they find? They found some things that might negatively surprise advocates of homeschooling. Home-educated graduates are the most likely to get divorced? Most likely to feel helpless in dealing with the problems of life? Most likely to lack any clear goals or sense of direction? Most likely to feel prepared for a vibrant religious and spiritual life?
Homeschool graduates are the least likely to be involved in political campaigns? Least likely to spend much time volunteering or going on mission trips? Protestant Christian school and homeschool graduates hold more strongly to the belief that morality is unchanging and absolute?
But before your columnist provides any more of these researchers findings, clouds the research waters, or is quoted out of context, several important points must be made. First, researchers must operationally define important terms in their studies. In this study, homeschooled was defined as a participant reporting he was homeschooled in high school. That is, the researchers only reported that the adults were homeschooled, Catholic schooled, public schooled, and so forth for some of their high school years; no information was given on whether they attended that type of schooling for two, eight, or twelve years. Further, the home educated were categorized as either religious or nonreligious, simply depending on whether their mothers attended religious services regularly. That is all that is known about these adults educational background.
Second, about two thousand randomly selected Americans, aged 23 to 49, were sampled and studied. This sample appears to have been planned and executed well. However, of those, fewer than ninety were home educated at all. And only some of these fewer-than-ninety had a religious homeschool mother during high school. This is a small sample size from which very few dependable conclusions can be drawn.
Finally, this studys findings directly contradict the findings of several other studies about adults who were home educated. For example, Ray found adults who were home educated to be more engaged in direct civic involvement than the general population of the United States. They were more involved in activities such as working for a candidate/political party/political cause, voting in national/state elections, and participating in a protest or boycott.2 Contrariwise, Pennings et al. state that adults from religious homeschooling were noticeably less civically involved on several measures than adults from several other categories of schooling.
Conclusions
Pennings and his colleagues have developed a valuable research concept. They are trying to understand to what degree the outcomes of Christian education match the motivations of adults who design and conduct the Christian education. It should be clearly pointed out, however, that this study has not . . . allowed for the establishment of quality benchmarks . . . (p. 36) regarding the thinking, religious beliefs, and actual behaviors of adults who were home educated. They might have quality benchmarks regarding those who attended state (public), Roman Catholic, and Christian schools but not for those who were homeschooled by Christian parents.
One might wonder: Do reports like this matter in the world in which we live? For better or worse, yes. Research is supposed to get at truth and accurate representations of reality. God says the following: By wisdom a house is built, and by understanding it is established; by knowledge the rooms are filled with all precious and pleasant riches. A wise man is full of strength, and a man of knowledge enhances his might, for by wise guidance you can wage your war, and in abundance of counselors there is victory.3 One can get wiser by reading and heeding true things but not from integrating skewed or too-limited representations of reality into his thinking.
Unfortunately, research like this often shows up in widely read publications. For example, the magazine Christianity Today reported on the findings of Pennings et al. and presented the culturally perceived positive and negative findings about the religiously homeschooled as if the data were reliable and properly comparable to data from other adults.
The authors of this study should have been very careful to explain the notable limitations of their study regarding the home educated. Their caveats should have been clear enough that others reporting on their study would find it difficult to misrepresent reality about any of the statistics in the study.
Caveat emptorLet the buyer bewarewhen it comes to reading and using research reports. Remember, the buyer or digester of researchers findings and conclusions must remember the . . . warning that . . . the goods he or she is buying are as is, or subject to all defects.4
Authors Note: Please feel free to send your questions about research related to home-based education and raising children to [email protected]
Endnotes:
1. Pennings, Ray; Seel, John; Van Pelt, Deani A. Neven; Sikkink, David; & Wiens, Kathryn L. (2011). Cardus Education Survey: Do the Motivations for Private Religious Catholic and Protestant Schooling in North America Align with Graduate Outcomes? Hamilton, Ontario, Canada: Cardus. www.cardus.ca, www.wnd.com/files/2011/07/110826cardus.pdf
2. Ray, Brian D. (2004). Home Educated and Now Adults: Their Community and Civic Involvement, Views About Homeschooling, and Other Traits. Salem, OR: National Home Education Research Institute, www.nheri.org.
3. Proverbs 24:3-6, ESV.
4. Retrieved December 12, 2011, from legal-dictionary.thefreedictionary.com/Buyer+beware.
Source...