A few years ago, my neighbor told me that Finland had the best primary education globally. I asked her why and she said I should search it up. It wasn’t until now that I asked myself: why do we compare education systems at all? Who determines which country’s education is more effective than another, and how reliable is it? 

After research, I found that the organization that ranks primary education systems worldwide is the Programme for International Student Assessment (PISA). Every three years, the PISA examines fifteen-year-old students’ ability to use their reading, mathematics, and science skills through standardized testing in more than ninety countries. In 2015, PISA began administering its tests electronically. Each country will average its test scores to represent its quality of primary education. These averages will then be compared with other countries for the best primary education systems. The test result will be published after every test. 

Every three years, PISA will release thirteen different test booklets with different topics and questions. Some of the topics in the reading section of the 2018 PISA test include cow’s milk and the Galapagos Islands. For reading, students are asked to read short paragraphs and answer questions. Their answers will either be available in the article directly or require interpretation. In the math section, students are tested on basic algebra, geometry, graph-reading, percentages, and ratios. For example, one question asks the students to calculate the distance between the center of a Ferris wheel and the ground. In the science section, students answer questions based on graphs relating to fossil fuels, volcanic eruptions, and more. There’s an additional section on the test called the global competence section, where students are quizzed on topics like ethical clothing and the rising sea level. Overall, PISA’s test questions resemble the SAT, the standardized test for college emissions in the U.S. The only major difference is that participating countries and economies choose the questions for the test, and PISA will determine if they are appropriate for use.  

According to the PISA results for 2018, the top ten countries and economies with the highest averaging reading, math, and science scores are China (578.7), Singapore (556.3), Macao (542.3), Hong Kong (530.7), Estonia (525.3), Japan (520.0), South Korea (519.7), Canada (516.7), Taiwan (516.7), and Finland (516.3). The United States (495.0) is placed at twenty-fifth. There’s a 62.4 point discrepancy between the 1st place (China) and the 10th place (Finland), which is 12.1 percent of Finland’s average score.  

Although PISA directly links students’ test scores to a country’s education quality, which is seemingly logical, it has many flaws. To put it simply, Valarie Strauss, a thirty-year education writer for the Washington Post, says PISA overly relies on standardized testing and has major flaws “with how some of the test questions are constructed,” “with how the sample of students are determined,” and “with how the tests are administered” in her article How PISA created an Illusion of Education Quality and Marketed it to the World (Strauss 2).  

This is true. First of all, the PISA test questions have a western cultural bias. This unintentionally suggests that countries that don’t align with Western education and ways of life have lower-quality primary education systems. Even though PISA claims that “the questions are reviewed by the international contractors and by participating countries and economies, and are carefully checked for cultural bias,” on their FAQ page, certain biases are unavoidable (PISA FAQ). For example, students in other countries, such as Saudi Arabia and Turkey, do not learn about the Galapagos Islands and Darwin’s Theory of evolution because it isn’t required in their school curriculum. This puts them at a disadvantage in the reading section of the PISA test, which could cause them to score lower. Their low scores do not reflect their ability to read, but rather their absent prior knowledge of the subject. Are Saudi Arabia and Turkey’s education systems less qualified than those who teach evolution? Who is it to say that? It’s all objective at the end of the day. 

This Western cultural bias phenomenon is also evident in the math section, where students are asked to calculate the distance between the center of a Ferris wheel and the ground. What if students in less developed countries have never seen a Ferris wheel? This may hinder their ability to answer the question. Therefore, their inability isn’t in math at all. Instead, it’s in the fundamental understanding of what a Ferris wheel is. Another math question asks about transferring photo albums from a memory stick, and the same argument can be applied. If the PISA incorporates more math questions that are solely algebra and geometry-based, then students will likely score higher on the test. And again, this is also true for the global competence section. Students in developing countries and rural areas may not understand the concepts of ethical clothing and avoiding fast fashion, which are things that people in first-world countries often prioritize to enhance sustainability. If students are from countries that don’t have enough clothes for everyone, then how are they supposed to know about avoiding fast fashion? Their curriculum, made by their government, will unlikely teach them about such issues. But does that make their education system inferior to a country that teaches this? Certainly not. The PISA test implies our first-world views onto less developed countries, which inevitably creates a Western cultural bias. At the end of the day, all the countries don’t just have one primary education system under a unified curriculum, so why are we using one objective test to evaluate all education systems?  

With that said, the PISA test results should be taken with a grain of salt, not just because of its Western cultural bias but also because the data collected from the test has multiple anomalies. For example, during the 2015 test, the response rate amongst students in Malaysia was 51%, which is significantly lower than the standard PISA response rate of 85%. Therefore, “the results might not have been comparable to those of other countries” (PISA FAQ 1). With evidence of response bias and a smaller sample size in Malaysia in 2015, it is certain that those results may not be accurate. Along with the Malaysian anomaly, in 2015, Argentina also faced challenges with sampling for the PISA test. That year, Argentina’s proportion of 15-year-olds declined, and coincidentally, they also restructured their secondary schools. This meant that PISA had limited options in choosing their sample students in Argentina. In this case, PISA states that “Argentina’s result might not have been comparable to those of other countries” (PISA FAQ 1). There are a couple more anomalies throughout the history of PISA tests, including one with the U.S: a test printing error caused the instructions to misdirect students, which affected their performance. These anomalies, suggesting PISA’s inaccuracy, are not listed below their test results where it’s visible to everyone. Instead, they are at the bottom of the frequently asked questions page, which most people don’t care to visit. The average website visitor will just look at the 2018 PISA rankings and assume the information is accurate. However, the information is not always accurate, provided these anomalies. Therefore, these flaws in the testing system, and the PISA's act of subtly disclosing them, ultimately jeopardize their credibility. 

The PISA test’s overemphasis on standardized testing puts the responsibility of a “good” education system on the students rather than sharing it with the teachers. PISA is indirectly reflecting students’ environments by looking at their test scores. Instead, they should also survey the teachers for a more accurate picture of the education systems. It is crucial to note that the quality of primary education is directly correlated with the teacher’s teaching style. Therefore, the quality of education should not only be reflected by students. In a Forbes article by Natalie Wexler titled What International Tests Can Tell Us About How to Improve Education, she introduces a study by Estonian education expert Gunda Tire. In Tire’s study, she uses a 2015 PISA survey and concludes that “teacher-led instruction was correlated with higher science scores, while student-centered approaches ––– for example, allowing students to design their own experiments ––– were correlated with lower ones” (Wexler 1). This is direct evidence that having a lower science score doesn’t necessarily equal a bad education system or a lack of resources. It can simply mean that the teaching style of a country’s education system is more focused on creativity and self-exploration rather than standardized testing. By simply looking at the test scores, PISA cannot distinguish this difference. For example, in the United States, high school teachers often give students the freedom to pick their own topics for science projects. This may explain why U.S students score lower on science than students from other countries with similar economic power.  

Another thing is, who is to say that teacher-led projects are better than student-centered ones anyway? Even though students who lead their own experiments may score lower on the PISA test, they will enjoy the curriculum more. Because, instead of constructing a double helix lab where students are only working for a grade, a student-centered project allows each student to research the topics that interest them. After all, who are we to say that using labs is a better teaching method because it yields better test scores? Self-guiding science projects allow the students to find what they love about the subject, rather than spoon-feeding them the curriculum, which will benefit them in the long run, even if current test results may not reflect that. And this is what the PISA doesn’t tell us by simply showing us the numbers. 

After China ranked number one on the 2018 PISA test, articles like China is No. 1 on PISA - but here’s why its test scores are hard to believe started appearing on the web. It was certainly hard to believe that China ranked number one. Every year, countless Chinese students leave their home country to attend schools elsewhere, and the fact that China is ranked number one for primary education just doesn’t sit right. Why would these Chinese students give up everything back home and leave their country for education? If China’s education is the best, then how come barely any American students leave to go there for school? If you dig deeper into how the PISA test is administered in China, then the answers are fairly simple. China is playing a game. 

On the 2018 PISA test, China selects the test scores of the four richest and most educated provinces: Beijing, Shanghai, Jiangsu, and Zhejiang (B-S-J-Z), to represent the country’s education quality. This gives them an unfair advantage because all the other countries have to average their test scores using all the states or provinces within the country. By only selecting four provinces for the PISA test, China’s score won’t be dragged down by the other twenty-seven poorer and less educated provinces. For Americans, that’s the equivalent of only taking PISA scores from states like Massachusetts, New York, California, and New Jersey and averaging them. Of course, the average scores will then be much higher. A study mentioned in the article The Strange Case of “China” and Its Top PISA ranking written by Mark Schneider, the director of the Institute of Education Sciences at the U.S Department of Education, shows that if the U.S combined test scores from “Massachusetts, Minnesota, Scarsdale, (New York), and say Evanston, Illinois, for the entire country, we would be right up there with B-S-J-Z China, with all attendant bragging rights” (Schneider 1). This alone shows that China’s education system is not strong at all. They’re just good at manipulating the system to make themselves look good. 

As Jay Matthews, a columnist for the Washington Post, puts it in his article Big Hole in Popular View that China beat the U.S in Education, “China’s position at the top (of education systems) inspires the widespread but mistaken belief that their schools are trouncing ours” (Matthews 1). By only having four provinces for the PISA test, China essentially ignores the poor, rural areas where education is barely accessible. In a study conducted from 2007 to 2013, evidence shows that the cumulative dropout rate in rural areas of China is “between 17 and 31 percent in junior high schools. Only half of the rural Chinese children went to high school, and only 37 percent of that group graduated” (Matthews 1). This is not what the world’s best education system should look like. Instead of implementing education reforms to uplift these rural communities, the Chinese Communist Party just forgets about them as if they don’t exist. At least in the U.S, students in rural areas have a better chance of graduating high school even though they are ranked below China on the PISA test. Letting China exclude the poor from participating in the PISA test, defeats one of their most important initiatives of having the “equity of learning outcomes for children, young people, and adults” (PISA Homepage). This hurts PISA’s reputation and points to its flawed testing system that allows countries to discriminate against the poor when determining the samples of students. 

What’s more ridiculous was that for the 2015 PISA test, China chose Beijing, Shanghai, Jiangsu, and Guandong as the four provinces, and had a reading score of 494, a math score of 531, and a science score of 518. In 2018, Zhejiang replaced Guangdong and the score went up to 555, 591, and 590. The substitution of one province boosted China’s score on the PISA test by more than 60 points per subject. China suggested that it was purely an “innocent coincidence” (Schneider 1). However, the detailed data of their 2015 and 2018 test performance by province “have not been shared” with the PISA countries (Schneider 1). Tom Loveless, a director at the Brown Center on Education Policy at Brookings Institute and a former Harvard policy professor, examined the PISA data from 2006 to 2015 with “497 observations” and concluded that the mean change in points for each country every three years is “9.5 points,” with a “standard deviation of 8.6” (Strauss 1). This meant that most nations’ PISA scores increased by around 10 points every three years, and it was highly unusual for China to gain around 60 points in the same amount of time. Therefore, by allowing China to cheat the system, PISA ultimately discredits other countries’ test scores, defeating the purpose of evaluating education systems and ultimately ruining their credibility.  

Even if China played a fair game and scored high on the PISA test, their test scores still wouldn’t reflect the actual educational quality. Yong Zhao, a University of Kansas education professor who grew up in Sichuan, China, said that China could “produce good test scores … but they have been struggling to make changes to their education” to make their students “more creative and entrepreneurial” (Matthew 1). This is true, and I can explain using personal experiences. Growing up in Shanghai, China, I attended one of the best elementary schools. My school imposed a serious culture and enforced a strict focus on academics. We were ordered to sit a certain way and stand a certain way. If anyone deviated from the rules, the teachers would criticize them and sometimes even asked them to sweep the bathrooms. After tests, teachers would read each student’s grade out loud, and those who scored the lowest would be asked to stand behind their desks for the rest of the class. The teachers would then lecture them about how they should be ashamed of getting bad grades. Students who struggled in class would be asked to forfeit their chance to go on field trips. Every day, students were expected to do four to six hours of homework after school.  

In fourth grade, a student in the front row lost focus in class, so after a warning, my math teacher lifted him out of his seat and tried dragging him out of the classroom. At the doorway, the student held on to the doorframe, and the teacher yanked him by the arm. The next day, that student came to school with a broken arm and a cast around it. Oh, and also, another student stabbed a teacher with a sharpened pencil because he felt academically pressured. Keep in mind that this was one of, if not the best elementary schools in Shanghai, China.  

When I began struggling in math class, the teacher asked my mother to send me to her house for private tutoring. When I went to the tutoring session for the first time, I saw six of my classmates, and that was when I realized why they were excelling in the class. The math teacher gave us mock tests during her tutoring sessions, which were almost identical to the actual tests in class. And with the expense of the tutoring fee, we were no longer subjected to bad grades. Years later, I realized that I didn’t understand anything in math class because the teacher purposefully skimmed through the material so that she could private tutor more students and therefore make more money. 

When I moved to the U.S in fifth grade, I learned that some of the events that happened at my school in China should’ve been outright illegal. So, how is China ranked number one for education quality on the PISA test? Even if they got first place without cheating the system, they don’t deserve it, quite frankly. This is because China’s education system does not care about students’ physical or mental health. And this leads to the question: do we really know if other countries’ PISA scores accurately reflect their educational quality? We will never truly know until we ask a student from another country. This is why PISA is using a flawed system to evaluate educational quality. 

After reading all this, you may ask, if the PISA is so flawed at determining the best education systems, then why are the tests still administered every three years? For some countries, a flawed comparison is better than no comparison. Perhaps by comparing education systems and quality, those countries who seek to reform their education systems can learn from those who rank high on the PISA list. Take Britain, for example. After the 2018 PISA test, British schools began using the Chinese education system to teach math. The article, Britain Turns to Chinese Textbooks to Improve Its Math Scores by Amy Qin reports that British educators believe “Shanghai … has the best math teachers in China.” Following this, the British government funded a “$54 million initiative” so that “more than half of the primary schools in England will adopt a teaching approach to math that is used in top-performing places like Shanghai” (Qin 1).  

As you can see, the PISA test and its rankings heavily influence the education policies and systems in countries like Britain. However, whether or not it’s a good idea for them to implement the Chinese education system is a debate for another time. The main point is, after seeing that countries like Britain still use the flawed PISA test to reevaluate their education, it is easy to conclude that countries compare education systems not only to learn from each other but also in hopes of placing higher on the PISA test next time for bragging rights. 

All in all, although the PISA test can gather large quantities of data and rank the countries by education qualities, its approach, from the creation of the test questions to the determination of the samples, to how the test is administered, is fundamentally flawed. By having a Western cultural bias in its questions, data anomalies, benefits to teacher-instructed classrooms, and allowing China to cheat the test, PISA is destroying its credibility. Therefore, the rankings should be taken with a grain of salt. With that said, the fact that countries are still following PISA’s ranking despite its flaws answers the question that countries compare education systems so they can score higher on internationally recognized tests like PISA. Ultimately, it is established that PISA’s standardized testing system cannot accurately depict the quality of education systems. So, then this question is: how should we really measure primary education systems around the world? Would it be by the students’ well-roundedness, intelligence, happiness, or something else?