Staff from across Columbia University Libraries and Information Systems (CUL/IS) gathered to present posters at the organization’s first assessment and service improvement mini-conference on June 20, 2013. Adrienne Garber and Tucker Harding, both educational technologists at the Columbia Center or New Media Teaching and Learning (CCNMTL), presented a poster on the Center’s efforts to use quantitative and qualitative data to develop Mediathread features.

Mediathread debuted at Columbia in Fall 2010 as an innovative, open-source platform for collaborative, multimedia analysis. This spring, Garber and Harding surveyed students and faculty from the 46 courses who used Mediathread during the semester to gather data on attitudes about the tool supported by observations of actual behavior. Their objective was to prioritize feature development for CCNMTL’s next round of design based on user needs. Their evaluation was guided by Design Research, an iterative cycle of research, development, and assessment used by CCNMTL to experiment with and improve innovative uses of technology.

Garber and Harding were excited to present their poster entitled, “Research-Based Design Decisions: Using Observational and Survey Data to Inform Tool Development.” The poster included research methodology, data collection and analysis, results, and evidence-based recommendations for future iterations of the tool.

The surveys provided insight into how students and faculty used the tool during the Spring 2013 semester to improve teaching and learning across various courses and assignment type. According to the student survey, close to 90% of Mediathread users would recommend that the tool be used again in future classes. Over 60% of students chose “Easy” or “Very Easy” to describe the ease of use of Mediathread.

“It is important for us as a Center to be critical of our own work and to make research-based decisions when designing our educational tools," said Garber. "We aim to find that 'sweet spot' of efficiency and effectiveness: to be cost-efficient with our resources while developing the most effective products for our faculty’s learning objectives in their courses. Rigorous analysis of user data pushes us closer to that goal.”

Garber and Harding used the survey results along with the observational data to make a series of recommendations for further developing Mediathread. For example, over half of student respondents said they were not provided a rubric - an explicit statement of the performance expectations and scoring guidelines for an assignment - for their Mediathread assignment. In conjunction with that finding, 1 in 5 students reported not having a clear idea of how their work would be evaluated in Mediathread.

This led Garber and Harding to discuss development beyond the code base and feature refinement, such as faculty support and training. The recommendations include establishing and describing best practices for using Mediathread as an educational tool, partnering with faculty on designing rubrics for assignments, and designing and testing assignments and exercises that would help students develop key skills for analysis and self-reflection using multimedia.

CCNMTL plans to gather longitudinal data of Mediathread over several semesters to better assess the value of Mediathread in attaining a course’s educational objectives.