Nine: Chart a Course

I'm so excited to share this project with you. It's been a long time since I felt like there was some magic in the air...and when The Muse came back, she did so with a vengeance.

This is a story about whether students who have similar paths (in the form of schools, teachers, and courses) experience similar outcomes (in the form of performance).

The Data
There are a lot of data packed into the representation of each student: gender, race, with/without disability, eligible/not eligible for free/reduced lunch, performance level for grades 3 - 7 on the state math assessment, teacher for grades 3 - 7. I built a template to pull these pieces from our student information system, but beyond that, I did not need to transform any of the data—no formulas, pivot tables, etc.

I did generate an alias for every student and teacher, because I fed their connections (which student had which teacher/course) into the Connect the Dots tool on databasic.io. I will explain the reason for this later on in this post.

The Build
The entire display is paper and glue, with map pins to attach each "student" to a board. That's it. The representations were built by quilling paper. I love that these data look fragile, but I have to tell you that the pieces are strong and resilient...much like our students. Also like our students, nearly all of them are unique.

Here's what you're looking at for a student. The star is made of 5 rays, representing the five grade levels (3 - 7). Grade 3 is in the 12 o'clock position, and the rest follow as you move clockwise around the star. If a student didn't attend one of our schools for a given grade level, there is no ray at that position.

The paper colour for each ray shows what the performance level was on the state math assessment for that grade. Orange is level one (well below standard), yellow is level two (below standard), light blue is level three (at standard), and dark blue is level four (above standard). Each ray is wrapped in a piece of coloured paper to indicate a particular teacher/school. We have six elementary schools serving grades 3 - 5, so there are six sets of colours: orange, purple, green, blue, red, and yellow. There are two middle schools serving grades 6 - 7, represented by blue and red. If a student attended one of our schools at a given grade level, but didn't take the test, the ray has the outer wrapper and is empty for the performance level.

On the inside of each star are four round disks of paper in white (w) and grey (g).

Note that we did not have any non-binary students at this grade level last year, so male and female were the only categories. I also collapsed our students of colour into one category, and while I strongly believe that this is not best practice, I am also legally obligated to protect personally identifiable information about students. At this grade level, we have some racial populations with only two or three students. When faced with a choice of leaving race out of the conversation completely or summarizing it into two categories...I chose Option B. Certainly, there are other data points I could have selected here, such as attendance or discipline. But I like that the center of the star includes some "internal" attributes that students bring with them, while the outer rays represent the "external" pieces in the form of teachers and system outcomes.

Here is a small group of students that shows the variations by missing grade levels, missing scores, gender, etc:

Now that I had 452(!) of these objects, I needed to figure out the connections between them. This is where the Connect the Dots tool helped. For each school, I uploaded a coded list of students and teachers. The online tool produced a network diagram like the one below, but more importantly, it provided me with a table showing me who was in each group, as well as the connections.

I selected the stars in each group and laid them out as a constellation. I placed stars that had teachers/courses in common closer together. Most of our elementary schools had 6 or 7 constellations. Here is one of them (before labels).


Each elementary school was organized on its own 24" x 36" cork board that had been covered in fabric. Stars were attached with map pins and then annotations were added to share what the group had in common. The six schools were organized into two sets of three, because we have two feeder patterns into middle school. This allows a look across the district.

Lessons Learned
When people use the phrase "labour of love," I will now know what they mean. I personally wove every piece of paper used in this display. I spent about 300 hours total building this story out of 3200 pieces of paper and a bottle of glue. Most of the work was done during my summer vacation: early mornings, late nights, weekends, at home and while visiting others. I drank a ton of coffee (mostly decaf, I promise). I got glue all over everything—my computer, my dining table, me. I have never devoted so much time and energy to creating one of these stories. But it was all worth it. This is the most beautiful display I've ever built. But beyond that, it is fascinating to look at. There is so much meaning packed into this. It captures and holds one's attention.

One of the things I have to remind myself is that when I uploaded the information about students and teachers, that was it—no information about gender, race, etc. And yet, it is obvious with some of groups that they have some of these attributes in common. In other words, the system either does or does not support certain types of kids. Now, this is not new learning, especially at a large scale level. But we are talking about small groups here. Let me give you one example. For students who take the advanced math track on one side of our district, one school only placed males there...the second nearly all males...but the third placed about even numbers by gender. What was most interesting about this third school is that nearly every girl who was in this track and successful in math had the same third grade teacher. That one teacher was the common denominator...and that group was different from every other in the district. And until I built this display, I would never have seen this. I was also surprised at how different every student appeared. Yes, I know that every child is an individual, but this display really drives it home. While there are a few (but not many!) students who had the same five teachers, they did not have the same internal attributes or the same outcomes.

If you'd like to see more pictures or learn more about our data stories project in our school district, please visit the companion page on our district web site.

Making Sense

Earlier this week, I had an opportunity to share three sets of data with building administrators. You've had a preview of some of the visuals: a new way to represent the achievement gap, cluster charts, and small multiples. The administrators had not, although I have shared lots of data in lots of other ways with them. This was our opportunity for some in-depth work.

With each round of data presented, I did a little bit of instruction so that they understood how to read the charts. Each table had some paper copies and a few focusing questions to get the conversation rolling. And then the real fun began.

I have never had a chance to watch people learn with new-to-them data representations. Bar charts, line charts, and scatter plots are commonplace. When you share these types of visuals, everyone already knows the drill. But hand people sheets with small multiples and a few of them will overlay the pages and hold them up to the light. Give them a set of line plots showing gaps among student groups, and they will spread them across the table to organize the pieces in different patterns. Hand over some cluster charts and watch and people fold the paper along various lines to build new learning.

It was absolutely fascinating.

As a way to gauge their engagement with the new charts, I tried the Talking Mats which were recently shared by Andy Kirk.

I provided three different colors of sticky dots, one with each round of data, and asked the administrators to place their dot when we transitioned to the next part of the workshop. In addition to watching them interact with the data---which was very powerful learning for me---the Talking Mats provided valuable feedback at the end. These are visuals for me, created by the audience, which have told me a lot about which charts gave the biggest bang for the buck. I'll know where to focus my time and work in the future.

A few people struggled with looking at the forest of data represented by small multiples, instead of the trees. Some made connections across the data sets. Still others wanted to argue with data because they were unnerved by what they were seeing for the first time. By the end, though, there was a very rich conversation about the meaning we'd been able to make from all of the data.

For the first time since I started this job, I had many public compliments about the work I shared. My favourite has been "Thanks for presenting data in a way that helps me learn." Here's to more opportunities to use Excel to support us in making sense of things.

Show the Data: Cluster Charts

In the last post, we explored the idea of adding bump(s) charts to our rotation of how we communicate our data. It's one way to show all of the data in a particular set. Another one I've been using quite a bit is a cluster chart. Full disclosure here, these are my own take on displaying data---a bastardized heat map, and certainly not based on heavy-duty math like real hierarchical cluster charts. So, really, I'm not sure what to call these...but in my current role, we're finding them to be very useful and I'm just rolling with cluster charts as my category.

This spreadsheet will eat your soul.

I get a lot of spreadsheets sent to me that look like the one on the right. I hate these with a fiery passion for a variety of reasons:

• Too much "ink" in the data-to-ink ratio. With all of those little boxes, I don't know where to look.
• And the colors. I feel like a circus came to town. But beyond that, the red and green are not particularly friendly to those with color vision issues...and I do work with some who are color blind. Are we really asking them to try and make decisions on student learning  based on this?
• Not to mention that all of the data is colored in. What's the point?
• And we have both numbers and colors. I'm not saying that you can't have both...or that they don't serve different purposes...but it's distracting. I'm constantly trying to make sense of the number patterns for each color.

I also think these data aren't useful because of the way they are organized. Alphabetical order is great for gradebooks, but not so much for trying to make sense of the data. Plus, we don't have any context---what if we're missing some signal in the noise? Suppose all of our low-performing students are boys...or in a minority group?

But let's say you are interested in showing both the progress students have made over time, as well as the characteristics of the students involved. We can reorganize the data by ranking the percentages on the second assessment (this is the "cluster" part). Then we can color code some additional information, such as gender or participation in a particular program. I also change the properties of the conditional formatting so that the fill and text are the same color, making the values seem invisible. Finally, I add thick white borders around all of the cells and resize the rows and columns. Here is a small part of the final product:

These are all the students who scored in the top level of our fictional grade 5 winter math assessment. Three of them improved a little, from light blue to a darker blue...others improved from lower on the scale (orange). But when we look at gender and program, another story emerges. Most of the students in the top category are female, not on free or reduced lunch, not in special education, and do not receive additional interventions through a Title I program.




See the difference when we look at students who have scored at the bottom for both fall and winter? Our population is mostly male and nearly everyone participates in one or more federal programs.

Maybe this representation doesn't necessarily hold any surprises, especially as we factor in free/reduced lunch. Children living in poverty typically do not perform as well as their peers. But one of the things I take away from this way to visualize that story is that we may need different interventions to support these students. Consider Student 39 on the right. He is receiving free or reduced lunch, special education and Title I services...and he's still ranked fourth from the bottom out of nearly 70 students. It doesn't mean that the school (and the student) aren't working as hard as they can. I do think it might mean that there are additional factors at work here that aren't (and can't be) addressed through the school. Perhaps the family is homeless or transient. Maybe the parents are going through a divorce...or the student has some medical issues. These are community-based issues and require different interventions to help close the gap for the student. I won't get up on my left coast soapbox about this right now. I'll just say that we have to work together on behalf of the whole child.

One of the pieces of information that is not represented in the visuals above is the number one item on teacher wishlists when it comes to reporting scores: progress. Sure, we have a bunch of students performing at the lowest level in the picture above, but that doesn't mean that they didn't make some growth.

This time around, I left the gender and program pieces coded the same, but I calculated the percent change between fall and spring and represented those in the leftmost column.

Look at Student 39 now. He's 12th from the top. Woo-hoo!

When we consider progress, we start to get a more equitable pattern---everyone is growing, and more often than not, it's our lowest performers who are making the biggest gains, even if they're still in the lowest part of the score breakdown.

By clustering similarly performing students together, either by scores or by progress, we get a much more useful pattern than we do with a spreadsheet that looks like a clown exploded on it. And, more importantly, we can show the data. In a very compact space, I can display everyone's scores and whatever demographic or program information is most relevant. And, I can fit the whole grade level on a single page.

I have no doubt that as we move forward, smarter people than me (I?) will continue to find new charts that help share everything we know about a group. Summary stats and charts will never go away---and they have their own purpose to serve. But sometimes we want the full version, not the Cliff's Notes. When we do, bump and cluster charts will be there.

Bonus Round
Want to see just how challenging colored squares can be? Play this online game. The rules are easy: just click on the one square that is different.

Hide and Seek

I want to circle back to an article I wrote a few years ago about my favourite data visualization.

Hierarchical Cluster Analysis by Alex J. Bowers from http://www.pareonline.net/pdf/v15n7.pdf

It shows all of the grades earned by students during their K - 12 journeys in two school districts. I love this chart because it finds a way to show all of the data in a dense, but succinct, format.

In The Visual Display of Quantitative Information, Edward Tufte states that Above all else, show the data. While the quote was applied to a different concept for visualizing data, when I look at the chart above, the quote rises to the surface of my thinking. Showing the data is no small task, and as educators, we spend a lot of time and energy not doing that. We summarize the data into neat little one letter grades or one number test scores. As teachers, we might see a set of scores...but we are the only ones to do so and we typically view them as numbers, not visual displays.Things hide in numbers and number sets.

But a recent paper shared in the Public Library of Science (PLoS) makes the case that things can be hidden in simple visuals, too.

CC-BY Weissgerber, Milic, Winham, Garovic

The authors of the article Beyond Bar and Line Graphs: Time for a New Data Presentation Paradigm assert that the ever-popular bar chart is a summary, and therefore "full data may suggest different conclusions from the summary statistics."  (It reminds me of Anscombe's quartet.) We often claim that pie charts are used to hide data. Et tu, bar charts?

I won't claim that the scatterplots and bump charts in the article are ground-breaking, but this paragraph in particular caught my interest (emphasis mine):

The infrequent use of univariate scatterplots, boxplots, and histograms is a missed opportunity. The ability to independently evaluate the work of other scientists is a pillar of the scientific method. These figures facilitate this process by immediately conveying key information needed to understand the authors’ statistical analyses and interpretation of the data. This promotes critical thinking and discussion, enhances the readers’ understanding of the data, and makes the reader an active partner in the scientific process. In contrast, bar and line graphs are “visual tables” that transform the reader from an active participant into a passive consumer of statistical information. Without the opportunity for independent appraisal, the reader must rely on the authors’ statistical analyses and interpretation of the data.

As educators, we might not view our work as a scientific process, but we must engage with our data. I feel pulled between the notion above that we may be oversimplifying our data presentations and some of the research about how an audience likes their data presented---which is typically charts that are the most familiar. This is not the Great Divide, mind you. We can bring these two things together with some education in the area of data literacy.

Or perhaps we underestimate our audience. I've introduced cluster maps, bump charts, and box-and-whisker diagrams to various groups this year. The first two required very little explanation. Box-and-whiskers did require a bit more orientation, but I never felt like the group using them struggled with the interpretation. I do think that concept of engagement between the visualization and the reader, as posed by the article is important. It's a different way to view interaction---a key piece of a good quality visual. It's not that the visual need be physically interactive...people don't have to be able to click, sort, or filter every chart---but we need to at least cause some thinking about what is presented.

After reading the PLoS article, I'm more convinced than ever that we need to when and why we share all the data. Bar and line charts may well be the fast food version of data viz, but we can begin to add to our visual diet by finding ways to show all of the ingredients.

Bonus Round
If you view the article on PLoS, you will have access to two Excel workbooks to help you make the charts presented in the article.

I'll share some of my own attempts to "show the data" in coming posts. Visit bump charts and cluster charts to learn more.

Big and Little

This material is reposted from my other blog. But I think it's worthy of sharing here, too. I like the idea of visuals which provide educators with both macro and micro levels of information. After all, school is not about making widgets. It's about learning with human beings. What are other ways we can "humanize" the data analysis process?

In the world of grading practices, there is standards-based...and hodgepodge. Hodgepodge grading (a term found throughout the research literature) refers to a score or final grade that represents more than learning. In other words, a teacher who assigns a grade that includes student "effort," whether or not the assignment was completed on time, neatness, or other factors in addition to what the student learned about the topic/subject/standard, is a hodgepodge grader. General consensus from the research is that this is not such a hot thing to do. Factors should be reported separately.

And then, there is this article which actually makes chicken salad out of these chicken sh..., er hodgepodge, grades. Here's the abstract:

Historically, teacher-assigned grades have been seen as unreliable subjective measures of academic knowledge, since grades and standardized tests have traditionally correlated at about the 0.5 to 0.6 level, and thus explain about 25–35% of each other. However, emerging literature indicates that grades may be a multidimensional assessment of both student academic knowledge and a student's ability to negotiate the social processes of schooling, such as behavior, participation, and effort. This study analyzed the high school transcript component of the Education Longitudinal Study of 2002 (ELS:2002) using multidimensional scaling (MDS) to describe the relationships between core subject grades, non-core subject grades, and standardized test scores in mathematics and reading. The results indicate that when accounting for the academic knowledge component assessed through standardized tests, teacher-assigned grades may be a useful assessment of a student's ability at the non-cognitive aspects of school. Implications for practice, research, and policy are discussed.

What's this all mean? According to the article, "25% of the variance in grades is attributable to assessing academic knowledge...and the other 75% of teacher-assigned grades appear to assess a student's ability to negotiate the social processes of school." Hodgepodge, indeed. However, "while administrators have indicated that they privilege standardized test scores over other forms of data (Guskey, 2007), little criterion validity has been shown for test scores as they relate to overall student school or life outcomes (Rumberger & Palardy, 2005), whereas teacher-assigned grades have a long history of predicting overall student outcomes, such as graduating or dropping out (Bowers, 2010)." So, if hodgepodge grades are better predictors for whether or not a student will finish school, why not find a way to use those to identify at-risk students?

And so the author (Alex Bowers) of the article did. And this, my friends, is one of the graphics:

Hierarchical Cluster Analysis by Alex J. Bowers from http://www.pareonline.net/pdf/v15n7.pdf

I won't get into the nitty-gritty here---you can read the article for yourself, if you like. But basically, what you're looking at here is every student's grades for their K-12 experience, for an entire district. The students are sorted/clustered according to the patterns their grades make. At the top, the kids who are mostly red in the heatmap are those who scored well throughout their academic career...at the bottom, the ones who struggled. The funky brackets on the left are used to cluster students with similar performance, the width and length of the brackets showing degrees of similarity. On the right, the black boxes include data which is not grade dependent, but was considered worthy of consideration.

Now, what the researcher found out from doing this sort of analysis isn't groundbreaking: Kids who struggle in school (gradewise) are more likely to drop out. And even though I can't condone hodgepodge grading, what is important here is that this is the first attempt I've ever seen that gets away from hodgepodge analysis.

I think every piece of research I've seen (up until now) does its best to mash the data into neatly digestible bites---just like we tend to do with student grades. Educational research doesn't represent individuals as individuals, but as populations. We seek to generalize, because we feel we have to, given the amount of data we collect. But I can't help but wonder what we'd see if we looked at all of the educational data we collect at both the micro and macro levels: the trees and the forest, like the graphic above. We are slowly taking steps to move hodgepodge away from the classroom performance level. Will we---Can we---see it disappear from the research, too?