Assessment

Data Carpentry values a culture of assessment, and we are consistently evaluating our workshops to understand the impact workshops have on our learners, and how we can improve our content delivery.

Dr. Kari L. Jordan, our Deputy Director of Assessment, leads our assessment efforts. She has established the Assessment Network, a space where those working on assessment within the open source/research computing community can collaborate and share resources.

Dr. Erin Becker, our Associate Director serves as Principal Investigator for our research efforts related to assessing the effectiveness of our workshops. This work is carried out under supervison of the Institutional Review Board at University of California, Davis. Our faculty sponsor at UC Davis is Dr. Megan Welsh, Assistant Professor within the UC Davis School of Education.

Please see our IRB proposal and determination letter for more details about our research efforts.

Surveys

Pre and Post-Workshop Surveys

We conduct pre and post workshop surveys for each workshop

Long-Term Survey

Additionally, we began collecting data on the long-term impact our workshops are having on both Data Carpentry and Software Carpentry learners. The Carpentry Long-Term Feedback Survey was launched in March 2017, and data is collected every six months.

Data

Data sets (provided as csv files) and assessment reports are in a GitHub repository, along with the version of the surveys that were in use at time of analysis.

Assessment Results

We will be releasing regular reports from our survey results. These reports are published through Zenodo and can be accessed through the links below.

Post-Workshop Reports

The analysis in this report serves the following purposes:

  • To inform the community of the impact Data Carpentry workshops have made on its learners.
  • To provide context for the survey responses as they relate to Data Carpentry learners.
  • To discuss what Data Carpentry is doing well, areas of improvement, and questions we should be asking.

The report finds that learners are reporting that after a workshop, their level of data management and analysis skills have increased, they have increased confidence in their ability to use these skills and that the learners have increased appreciation for these skills (i.e. scripting) to improve and promote reproducible research. 95% of learners agree or strongly agree that they would recommend the workshop to a colleague.

Long-Term Survey Reports

The long-term survey assessed confidence, motivation, and other outcomes more than six months after respondents attended a Carpentry workshop. Provided below are a few highlights from the data.

  • 77% of our respondents reported being more confident in the tools that were covered during their Carpentry workshop compared to before the workshop.
  • 54% of our respondents have made their analyses more reproducible as a result of completing a Carpentry workshop.
  • 65% of our respondents have gained confidence in working with data as a result of completing the workshop.
  • 74% of our respondents have recommended our workshops to a friend or colleague.