Big and Little

This material is reposted from my other blog. But I think it's worthy of sharing here, too. I like the idea of visuals which provide educators with both macro and micro levels of information. After all, school is not about making widgets. It's about learning with human beings. What are other ways we can "humanize" the data analysis process?

In the world of grading practices, there is standards-based...and hodgepodge. Hodgepodge grading (a term found throughout the research literature) refers to a score or final grade that represents more than learning. In other words, a teacher who assigns a grade that includes student "effort," whether or not the assignment was completed on time, neatness, or other factors in addition to what the student learned about the topic/subject/standard, is a hodgepodge grader. General consensus from the research is that this is not such a hot thing to do. Factors should be reported separately.

And then, there is this article which actually makes chicken salad out of these chicken sh..., er hodgepodge, grades. Here's the abstract:

Historically, teacher-assigned grades have been seen as unreliable subjective measures of academic knowledge, since grades and standardized tests have traditionally correlated at about the 0.5 to 0.6 level, and thus explain about 25–35% of each other. However, emerging literature indicates that grades may be a multidimensional assessment of both student academic knowledge and a student's ability to negotiate the social processes of schooling, such as behavior, participation, and effort. This study analyzed the high school transcript component of the Education Longitudinal Study of 2002 (ELS:2002) using multidimensional scaling (MDS) to describe the relationships between core subject grades, non-core subject grades, and standardized test scores in mathematics and reading. The results indicate that when accounting for the academic knowledge component assessed through standardized tests, teacher-assigned grades may be a useful assessment of a student's ability at the non-cognitive aspects of school. Implications for practice, research, and policy are discussed.

What's this all mean? According to the article, "25% of the variance in grades is attributable to assessing academic knowledge...and the other 75% of teacher-assigned grades appear to assess a student's ability to negotiate the social processes of school." Hodgepodge, indeed. However, "while administrators have indicated that they privilege standardized test scores over other forms of data (Guskey, 2007), little criterion validity has been shown for test scores as they relate to overall student school or life outcomes (Rumberger & Palardy, 2005), whereas teacher-assigned grades have a long history of predicting overall student outcomes, such as graduating or dropping out (Bowers, 2010)." So, if hodgepodge grades are better predictors for whether or not a student will finish school, why not find a way to use those to identify at-risk students?

And so the author (Alex Bowers) of the article did. And this, my friends, is one of the graphics:

Hierarchical Cluster Analysis by Alex J. Bowers from http://www.pareonline.net/pdf/v15n7.pdf

I won't get into the nitty-gritty here---you can read the article for yourself, if you like. But basically, what you're looking at here is every student's grades for their K-12 experience, for an entire district. The students are sorted/clustered according to the patterns their grades make. At the top, the kids who are mostly red in the heatmap are those who scored well throughout their academic career...at the bottom, the ones who struggled. The funky brackets on the left are used to cluster students with similar performance, the width and length of the brackets showing degrees of similarity. On the right, the black boxes include data which is not grade dependent, but was considered worthy of consideration.

Now, what the researcher found out from doing this sort of analysis isn't groundbreaking: Kids who struggle in school (gradewise) are more likely to drop out. And even though I can't condone hodgepodge grading, what is important here is that this is the first attempt I've ever seen that gets away from hodgepodge analysis.

I think every piece of research I've seen (up until now) does its best to mash the data into neatly digestible bites---just like we tend to do with student grades. Educational research doesn't represent individuals as individuals, but as populations. We seek to generalize, because we feel we have to, given the amount of data we collect. But I can't help but wonder what we'd see if we looked at all of the educational data we collect at both the micro and macro levels: the trees and the forest, like the graphic above. We are slowly taking steps to move hodgepodge away from the classroom performance level. Will we---Can we---see it disappear from the research, too?

When Full Brains and Calendars Collide

This is not going to be a post about why I haven't been posting. Okay, maybe a bit of it will be.

I've been putting together an article on using data visualization as feedback in the classroom, and my "free time" has been pulled in that direction. While I'd love to share the content here, I can't until a decision about publication has been made. But I can share a few general observations about what I've read.

• In education, we've known for a long time (since 1986) that using charts and graphs as feedback is second only to identifying interventions for raising student achievement.
• Almost all of the research about using visuals as feedback in the classroom has been done in classes involving computers: engineering, online classes/schools, programming, etc. Those with access to technology have been the ones able to harvest data and consider its applications. But, there's not a lot out there for general K-12 classrooms, other than using visualizations with special needs children.
• There are almost no examples of applying "business intelligence" models to K-12. Keep in mind that I would never advocate for schools to run as businesses, but what the private sector has learned about applying an understanding of visual perception to charts and graphs runs circles around anything in education. We need to raid and pillage this knowledge.

Summary: We know using data visualization is a powerful tool for the classroom and student learning, but we sure don't seem to make any effort to use it.

After reading a huge stack of educational research and digging deeply into several books on data viz, my head is bursting with ideas. If you have to have a problem in your life, then this is very best one to have. I highly recommend this sort of mania. However, with approximately two weeks out of this month spent on the road, this has also been one of the most frustrating problems to have: No time to write or finish my half-cooked workbooks for this space. I want to pull my hair out.

But my travel will be done in a week...and then I can have some time and space to pursue these ideas and beat them into submission.

In the interim, I have a post from my other blog to pull over and show you here. Wherever you are, I hope you're learning lots, too.

Gradebook...in...Space!

When I'm out and about with educators, I always like to ask what they need their data to do. Shiny software can be great, but all too often, I hear from teachers and administrators that they're stuck with pre-packaged analysis and don't have the flexibility to do what they want. Using Excel might not fill in all of the gaps, but I like to think about the ways it can answer the questions educators have.

After prompting a discussion about what they'd like to see, a teacher talked about how she'd really like to overlay her gradebook onto the student seating chart. I thought that was a great idea. We often look at classroom data through the lens of time, but we don't often see it in space.

This exchange happened a couple of years ago, and I finally sat down to develop the idea. In this post, I'll share the most basic version; but there are plenty of additional options to add on. I started by modifying the Beginner's Gradebook to include 30 students, then drew a very traditional seating chart on another worksheet. I filled in students names below each "desk." Keep in mind that you could make any arrangement that you want with the spreadsheet: a different number of desks, four students at a table, horseshoe arrangement, the remaining room set up (with the one kid you always need to have next to your desk). Lots of possibilities.

Next, I highlighted the range of assessments on the Grades sheet. I named the range "Assessments." I'm creative like that.

On the sheet with the seating chart, I created a data validation list using the Assessments range. This way, I could display different data on the seating chart. (Need a refresher on creating the data validation? Watch here.)

Then, I added an INDEX/MATCH equation in the "desk" cell for every student. Here is the one for Cathy Andrews: =INDEX(Grades!C8:AJ8,MATCH(T3,Assessments,0))  The formula tells Excel to look on the spreadsheet with the Grades, in the range of assessments (columns C - AJ) for Cathy Andrews (row 8), and match the score for the assessment on the Grades sheet with the one selected in the dropdown list (T3).

I would like to find a different way to do that part. Having to individually list which kid matches each row of data is not particularly friendly. And, when you move kids around in the room, you'll need to remember to move their equations with them. Since you can't have the formula MATCH on two variables (name of student, title of assessment), I've done it the cleanest way I can think of...but I'll keep hunting. If you have a solution, I'd love to hear it and hope you'll share in the comments of this post.

Finally, I added some conditional formatting to the "desk" cells.  The end result is a heatmap'ish thing like this:

It's kind of fun to play with. I did manipulate a few things in the data as discussion points. Does Dorothy Gale's performance second quarter catch your eye? Would you call home to find out? Are there some areas of the room outperforming others? Do you think something would change if you spent more time there or should you regroup students?

You can download the sample gradebook here. I'd love to hear your ideas about how to use this type of tool. I'm thinking about ways to incorporate attendance data...or graphs. What would you want to see from a birdseye view of your classroom?

Bonus Round
Are you attending the ISTE conference in June? Come hang out in my workshop. We'll play with Excel and other tools...and I'd love to hear more of your ideas about what would be useful data analysis for you.