General

Q: What ages/levels is Test Your Culture appropriate for?

A: Test Your Culture works for all ages and knowledge levels, whether you’re in elementary school or a college professor.

However, it tests principally adult cultural knowledge, with few children’s interests. Therefore, if a child takes it yearly, say at ages 5–10, it would reflect growth only in their “adult” knowledge of things like Top 40 artists, mainstream movies, and politicians — and not the universe of TV shows or music appropriate for their age.

Scores

Q: Why do you calculate the final score out of the top 10,000 cultural items, instead of out of all items?

A: We don’t calculate the score for the entire range of items you know (out of the 1,000,000+ cultural items in Wikipedia), because it isn’t possible to do so accurately on a short quiz, due to how sparse knowledge quickly becomes.

For example, suppose that at the 10,000th most popular item, you have a 1-out-of-10 chance of knowing it. We can still measure that you're checking 1 out of every 10 checkboxes, and develop a decent estimate. But once you get to rarer items such as the 100,000th most popular, knowledge drops off steeply, so you might only have a 1-out-of-100 chance of knowing it — which means we’d need to ask you several hundred items to measure accurately. To keep the quiz a reasonable length, we cut it off earlier.

Q: Why do you calculate the final score out of the top 10,000 cultural items, instead of a different number (like 1,000 or 100,000)?

People’s knowledge profiles vary in both breadth and depth. (Think of people familiar with the top musical artists in all genres, compared to people who know every heavy-metal band but nothing in any other genre.)

If we reported the number of known items out of only the top 1,000, we’d mainly be measuring breadth, without an opportunity for ‘deep’ people to shine. And if we reported out of the top 100,000, it turns out that depth would totally overwhelm the number, leaving ‘broad’ people left out. (It’s also much more difficult to measure, as discussed in the previous question.)

It turns out 10,000 is a ‘sweet spot’ which strikes a reasonable balance between both breadth and depth, so both types of people can shine (and people with breadth and depth will shine even more).

Q: Why is the quiz different each time I take it? Why does my score change slightly?

A: In order to estimate how many items you know out of the top 10,000, we sample 145 randomly-chosen items out of the full 10,000 to produce a final estimate, which has a margin of error. This is similar to how political polling works — when a news organization calls 1,000 random households to ask how they’ll vote and then extrapolates that to the whole country, it includes a margin of error the same way.

The only way to determine exactly how many items you know would be to test you on every single one of the top 10,000 — and we assume you have better things to do with your time!

If you want a more accurate test result, it’s scientifically valid to take the quiz as many times as you like and average together the scores. However, because this has diminishing returns, so don’t recommend doing it more than a few times.

Score comparisons

Q: How does my score compare? What does the percentile mean?

A: In addition to your score, you receive a percentile which lets you know how you compare against other Americans aged 18+. A percentile means you know more than that percent of the population — e.g. being in the 20th percentile means you know more than 20% of Americans, while being in the 80th percentile means you know more than 80% of Americans. Percentiles range from 0 to 99, and anything 50 or above means better than average.

Q: Is my percentile calculated against people who have taken the quiz, or the American population overall?

A: Short answer: the American population overall (aged 18+).

Long answer: we can only calculate percentiles from people who have taken the test (and filled out the associated survey), but these people aren’t necessarily representative of the American population overall (e.g. they tend to skew younger and more-educated).

To address this issue, we include a survey at the end on participant demographics. Then, by comparing with US Census data, we normalize our population of test-takers to match the US population in distribution of age, gender and education level. This normalization allows us to produce percentiles which reflect the country as a whole.

Note that while your score will not change if you re-visit your result page weeks or months later, your percentile may change slightly. This is because we re-calculate percentiles over time to become increasingly accurate as we get more data, especially quiz results from those with less education and those who are older.

Q: What about other comparisons, like my percentile by age, or percentiles for non-Americans?

A: As soon as we have enough data we plan to make percentiles by age available too. After all, an 18-year-old might have a low percentile for the general population, but a high percentile for 18-year-olds.

We’re still determining how best to produce comparisons for non-Americans, since there’s so much variation both between and within countries, and especially between English-speaking countries and non-English-speaking countries. But in the meantime, compare your results with friends to see how you stack up!

Quiz items

Q: What categories of items does Test Your Culture include? Why these and not others?

A: Test Your Culture includes specifically:

  • People from all areas (entertainment, politics, history, etc.)
  • Films and TV shows
  • Music groups, albums and singles
  • Books and short stories
  • Plays, musicals, operas, and music compositions
  • Artworks (paintings, sculptures, etc.)
  • Video games
  • Comics and anime/manga

In an effort to keep the quiz as “family-friendly” as possible, we attempt to filter out adult film stars and people who are principally either perpetrators or victims of violent crime lacking in historical significance.

Notable categories we don’t include are places, organizations of people with the exception of music groups (e.g. companies, sports teams, political parties), products, or events. Ultimately we feel these belong more to ‘society’ broadly than ‘culture’ specifically. We also exclude fictional characters, on the assumption these are already accounted for in their fictional works. But we’ve got big plans for including some of these categories in future quizzes, so stay tuned...

Q: Test Your Culture is America/English-centric, you hegemonic cultural imperialists! Why isn’t it more global?

A: Your score is based on your knowledge of the top 10,000 cultural items, as ranked by their popularity on the English-language Wikipedia — which means they’re determined mainly by US popularity, with some items boosted as well by interest in the UK, India, Canada, Australia, and other English-speaking countries.

We’d love to produce unique English-language quizzes for different nationalities — so Brits would see more BBC shows and Indians would see more Bollywood stars. Unfortunately, Wikipedia doesn’t separate out their page traffic data per-country, so we’re unable to do that at the moment. But if/when they do, it’ll be at the top of our list.

As for producing the quiz with non-English content in other languages (like French or German), we’d also love to if/when we have the resources. It takes a lot of work to map Wikipedia articles into our desired categories for any given language, but it’s definitely on our radar.

Q: I'm American, and there are a bit too many international soccer players (“footballers”), British royal family members, Bollywood actors, and cricketers for my taste. What gives?

A: Because our popularity rankings are based on English-language Wikipedia, this also includes traffic from the UK and India — hence items like these which are particularly popular there.

However, rankings are still primarily based on American traffic. And since the main value of your result is in comparing with your friends or the overall population, your score will stay the same relative to other Americans — that is, unless you have a friend who’s an expert in Bollywood cinema. ;)

Q: How often are quiz items updated?

A: Quiz items are updated monthly from Wikipedia. This includes both item information (descriptions and images) and item rankings (based on traffic over the past five years).

Data comes from a snapshot of Wikipedia produced at the start of each month. Because it takes a few days for Wikipedia to export the data and then another few days for us to import and process it, updated items may not appear until the middle of the month.

Q: Are quiz items ordered by popularity?

A: No. In any given step, the first column (or first third) of items is more popular in general than the last column (or last third), but individual items are not in any strict order.

Q: There’s a mistake in one your quiz items. Can you fix it?

A: We’d like to! Please e-mail us if the mistake is one of the following:

  • A quiz item that doesn’t belong (e.g. we’ve listed an event as a person)
  • A mistake in the short item description listed directly below each item’s name (e.g. “author”)
  • A mistake in the pop-up item summary that looks like gibberish or like part of a sentence is missing

However, if it’s an editorial mistake in the pop-up item summary (you disagree with the information or think the wording needs improvement), please follow the link to Wikipedia and correct it at the start of the article there. Since we import content from Wikipedia on a monthly basis, it may take until the middle of the following month to be reflected here.

Survey

Q: Why do you ask survey questions on birth year, gender and nationality?

A: We ask about birth year so we can measure the extent to which people’s cultural knowledge changes over time. But all three are also to write blog posts that answer fun questions like what cultural items are more or less known by different groups of people.

Q: Why do you ask Americans about their highest level of education and their ZIP code? Is this for marketing?

A: This is not for marketing — we don’t sell any data, and our quiz is 100% anonymous. We ask about education to compare with US Census data in order to measure how test-takers differ from the general US population, and ZIP codes are so we can produce blog posts with fun statistics and maps on cultural items which are more or less known by different areas of the country.

Q: In the survey question on nationality, why do/don’t you list ‘X’ as a nationality? That’s offensive because it’s generally accepted that ‘X’ isn’t/is a country!

A: Trying to make geopolitical decisions on what is or is not a country is waaaaay above our pay grade, so we just stick to the internationally standardized list of ISO 3166-1 codes. For example, Puerto Rico is included as its own nationality even though it’s part of the United States. Similarly, Hong Kong and Taiwan are listed separately as well, regardless of their relationship to China. We’re not taking a political stance and don’t mean to offend anyone.

Method

Q: How do you calculate the relative popularity/ranking for each item, in order to determine the final 10,000 the result is based on?

A: The popularity of each item is based on the proportion of internet traffic to its English-language Wikipedia article.

Because we seek to produce a ranking of “general” cultural knowledge as opposed to what’s popular just today or this month, we use a rolling average of an item’s popularity over the past 5 years. To avoid the effects of sudden spikes of news or other artifacts, we exclude the top 5th percentile of highest-pageview months. Combined, we find these produce an overall ranking of cultural items which feels intuitively reflective of broad popularity over time.

Q: What is the math behind Test Your Culture? How do you calculate the final result?

A: It turned out to be more complicated than we’d expected! When we put together Test Your Vocab, the math was fairly straightforward because the people’s vocabulary patterns tend to be fairly straightforward: they usually know all words up to a certain point, their knowledge gradually drops off, and the number of English words in general usage is finite (around 45,000). Therefore, we could focus on finding the ‘zone of transition’ and making an accurate estimate based on its center. But when it comes to cultural knowledge, three factors complicated things.

First, unlike vocabulary, people generally have big gaps in their knowledge even of extremely popular items — e.g. you might know politics and history but have no idea who the current top teen pop star is, or vice-versa. This means we can’t rely on a large zone where people know all the words, but rather always have to test the whole range.

Second, people’s cultural knowledge goes very deep but also becomes extremely sparse — you’ll still know some items at rank 100,000 or even 1,000,000. So we can’t rely on finding the center of a ‘zone of transition’ because there isn’t a center at all, just a gradual decrease that never ends.

And third, people have different ‘shapes’ of knowledge. Some follow a ‘broad but shallow’ shape (foxes who know all the popular things but don’t go in-depth) while others follow a ‘sparse but deep’ pattern (hedgehogs who only know one or a few areas, but really know them). We needed a method that would work for both styles, and anywhere in between.

So the short answer of what we did was to discover the formula for the mathematical ‘curve’ which accurately these shapes of knowledge, and then we find the particular shape of each person’s most likely ‘curve’ based on their answers. We then apply this curve to the full set of 10,000 items to estimate how many they know in total.

Or, for those of you who speak data science: we found a successful model for response probabilities to an item of given rank, based on parameters which correspond essentially to the test-taker’s breadth and depth of knowledge. Based on binary answers to a given set of ranks, we perform binomial regression via hill climbing on maximum likelihood estimation to determine the most likely parameters for the test-taker. We use active learning across multiple steps to maximize accuracy, and use the model’s cumulative distribution function to calculate the total estimated number of known items out of the first 10,000.

Data

Q: Is your database of ranked cultural items available?

A: It’s not currently available (and would take additional effort to make it so), but we’re open to exploring it if it makes sense. If there’s something you’d like to do with it, please e-mail us and let us know what it is, and whether it would be for personal, educational, academic/research, or commercial use.

Q: Can you share the raw response data?

A: If you’re a student or researcher who’d like to analyze the set of anonymous response data, we’d be happy to share — and even link in our blog to any interesting results you might find. Please just e-mail us.