Ideas for a podcasting experiment

I came back from the 05 AECT conference in Orlando fired up with the idea of doing a pedagogical experiment involving podcasting. Here’s my inital thoughts:

Podcasting Experiment

I have anecdotal evidence that having students do the process of reading aloud what they have written in order to record it as a podcast improves the quality of their writing style. This is most noticeable with students who put least effort into their writing and who produce stream of consciousness writing with little attention to grammar or punctuation. If this is indeed the case, then one should be able to measure real improvement in quality of writing where student podcasting is used in an experiment.

One could investigate this phenomenon in a number of different dimensions:

1. Longitudinal. Does writing quality improve and ‘stick’ over the long term? Can one use podcasting in a freshman writing class (e.g. ES, IP) and see whether writing ability improves and stays good until graduation. Would regular use of student podcasting maintain higher writing quality? Can students of poor writing ability be motivated over the long haul to improve their writing?
2. Latitudinal. Can we demonstrate an effect of podcasting over different classes and with different initial writing abilities?
3. Across colleges. Can a ‘podcast writing’ effect be discerned in different college environments?

What sort of experiment could be conducted to investigate one or all of these factors? Given the technical impediments to mass podcasting by students the best hope would be seek changes in writing technique within a single semester and a single class; obviously a pilot study would be required. Moreover, the technique needs to be well scaffolded within the content of the course being taught, that is, it’s not likely that there would be an observable effect if all that is done is to have students make a single podcast at the beginning and end of the semester with no use of podcasting in between. With this in mind, perhaps the best kind of course would be one which already involves the use of ‘technology’ in it’s subject matter but which also includes a strong writing component. Indeed, if the student’s writing were presented in their own blogs then they could not only host their own podcast but also it would make access to their work a lot more straightforward [rephrase]. But what of teaching style and pedagogic environment? These issues might be addressed in a study which is carried out simultaneously at two different colleges.

Use of iPods as an experimental variable.

The study would need to be designed to facilitate quantitative statistical analysis; to this end the following treatments could be applied:

1. a group of students not podcasting and not having an ipod (the ‘control’)
2. a group of students who podcast but who are not given an ipod
3. a group of students who podcast and who are given an ipod to use with the podcasts

[The above would enable us to consider whether podcasting on it’s own was effective or whether the incentive of listening to one’s own and classmates’ work would have a greater effect. ]

and the following factors could be examined:

• initial writing ability. Is there a bigger effect with poor writers coming in to the class than with better writers?
• cross college influences

The effect of instructor grading and assignment differences would be controlled for by :

1. Agreeing on a grading rubric to assess the writing by
2. All teachers grade a range of paper(s) — good, medium & bad — to establish a grading baseline
3. Since the writing will be presented in blogs it would be an easy matter to randomise the assignments that each prof grades. Thus each prof would grade some assignments from the other college. The teacher may or may not decide to use this particular grade in their final assessment.

Other requirements:

1. A reasonable class size so that there are an acceptable number of students per treatment. Given the variation in writing ability at 10 students per treatment per class would be needed, so a minimum of 30.
   [There’s also the issue of how & where one would obtain at least 10 ipods for the students to use during the semester and how to handle those students who don’t get them]

Possible Procedure

1. Select two classes with similar topics (mine would be something like ‘Impact of the Internet on Society) to run in the same semester (but start and finish dates don’t have to coincide exactly).
2. Agree upon a grading rubric and the teachers grade some papers of varying quality prior to the course start to assess a grading baseline.
3. Decide the number of writing assignments to be used — three would probably be optimal with one at the beginning of the semester, one in the middle and one at the end.
4. Decide the general content of the writing assignments so that the expectation is for a similar length of composition (the assignments need not be exactly the same across schools — or need they?)
5. After the first class session when the total number of students in each class is known, randomly allot students to each of the three treatments outlined above.
6. The initial writing assignment will serve as the ‘baseline’ for progress — this should be submitted before the podcasting takes place. Teachers grade the number of assignments corresponding to the number of students in their class but picked randomly from the total pool of students from both classes.
7. Have students do an agreed number of podcasts throughout the semester and integrate this within the course (but what about the control group with no podcasting?).
8. At mid-semester grade the mid-point assignment in the same way. This writing assignment should also be podcast.
9. At the end of the semester, grade the final assignment which is also podcast.
10. Finally, administer an online survey examining student’s perceptions of the podcasting process and their perceptions of effect on their writing.

Outcomes

Outcome	Statistical test
Does student’s writing improve with podcasting?	Analysis of Variance on final-initial grade
Do poor students improve their writing more than good writers do?	Use initial baseline grade as a covariate in ANOVA. Regression analysis of initial grade vs improvement. Or group students into 2 or more prior writing ability buckets and analyse accordingly
Does the college environs have an effect?	Compare ANOVA treatments across campuses
What’s the ipod ‘technology’ effect? Does having the ipod make a difference	ANOVA with the two treatments. Look for differences between them

The experimental design should account for differences in grading, teaching style, and course material so that these are ‘controlled out’ and we measure the treatment effects only.