EDLI Program Evaluation Plan

Overview: With a central mission to enhance the teaching and learning effectiveness in the online or multimodal settings at MSU, it is of critical importance to develop a sound plan for evaluating what and how the EDLI team has been and is building towards achieving its vision as well as the core teaching and learning mission of the university. Therefore, we have examined the contexts and operations of the EDLI projects and activities and developed the framework, plan, and process for the EDLI program evaluation. 

Outcomes: We established the foundation for guiding and implementing the EDLI program evaluation. We presented the design, process, and existing findings from the EDLI program evaluation at the annual conference of the American Evaluation Association.

In the wake of the COVID-19 pandemic and the growing national focus on education, recent research based on 897 faculty and administrators at 672 U.S. higher-education institutions reported that almost all reporting institutions transitioned to emergency teaching and learning approaches via online or hybrid delivery. Further, all faculty pivoted to online teaching (Johnson et al., 2020). As a result, those without any prior experience in online teaching faced a significant challenge to learn how to teach online. At MSU, the Enhanced Digital Learning Initiative (EDLI) was formed around these critical needs of the pandemic in summer 2020. EDLI is essentially a collaborative effort of digital pedagogy experts and researchers from three colleges of MSU, in collaboration with MSU IT and MSU Libraries, designing and delivering faculty professional development programs to improve the teaching and learning effectiveness in online or multimodal settings.

The assessment of the effectiveness of faculty professional development programs is increasingly important in higher education to not only establish credibility internally and externally but also to improve program operation. However, such assessment is difficult to accomplish, as a professional development provider impacts the eventual learning outcomes indirectly through those faculty who implement or are expected to implement relevant teaching interventions, and directly influence students’ learning outcomes.

Given the unique cross-college context and constraints that the EDLI is situated in, Jun has led the EDLI team to develop the framework, plan, and process for the program evaluation of the EDLI. The EDLI program evaluation plan aims to (a) assess the program theory: How, in theory, does the EDLI achieve its intended change, and b) evaluate the effectiveness of the EDLI faculty professional development programs.

To assess the program theory, we developed a logic model for the EDLI that helps articulate the different component tasks and their interrelationships under the umbrella of the initiative; it provides an understanding of what and how the EDLI team has been and is building towards achieving its vision and the core teaching and learning mission of the university. To evaluate the effectiveness of the EDLI faculty professional programs, we developed an evaluation process that includes the following five levels of program evaluation: Level 1: Participation, Level 2: Satisfaction, Level 3: Learning, Level 4: Application, and Level 5: Impact. Each level corresponds to different sets of evaluation questions. 

In terms of the data collection for program evaluation, we use a variety of methods and assessment tools such as attendance records, surveys and interviews to gather feedback on participant expectations, participants’ portfolio artifacts, and data that are gathered to answer the program’s impact, which are based on measures for proximal, intermediate, and/or distal outcomes of a program project. The following is an evaluation report (at Level 4. Application) produced:  

In April and May 2021, Jun, Jeremy, and Sarah conducted a focus group session and interviews with a total of 6 faculty members at the Broad College of Business. These faculty members are among those who have been supported by EDLI’s programming (such as supporting their transition into multimodal teaching during the pandemic), or have collaborated with EDLI on different initiatives (such as incorporating self-reflection for enhancing teaching) during this past year. These focus group session and interviews aimed at addressing the following key evaluation questions:

  • What are the faculty perspectives on technology integration before implementation of relevant technologies? Is there any change in technology integration after implementation?
  • Which factors affect faculty’s implementation of technology integration? Is there any barrier to technology integration?
  • How do faculty develop knowledge of technology integration, multimodal teaching, or self-reflection for enhancing teaching?

The focus group discussions and interviews were recorded via Microsoft Teams and then were transcribed verbatim. The transcripts were analyzed using constant-comparison analysis (Dye et al., 2000). The results adequately reflected the evaluation questions.

1) Faculty’s perspectives on technology integration

Faculty perceived the difficulty in meeting the needs of prompt transitioning to online or multimodal teaching and the amount of effort and learning required of teaching online during this past year. 

“My classes were frustrating really from the beginning, but I actually have the opposite experience because my class is really not interactive and with so much material to cover in such a short period of time.” ~ Focus group session, April 12, 2021

Faculty perceived the relevance and usefulness of relevant professional development workshops/training provided for them to transition into multimodal or online teaching during this past year. 

“I did a couple of guided SOIREE (Spartan Online Instructional Readiness Educational Experience) and another one, which hopefully you know about, ____ (EDLI team member name), since you’re the one that led both of them. And without, which, by the way, this last year would have been a complete and utter disaster and failure. So thank you.”  ~ Focus group session, April 12, 2021

2) Factors influencing faculty’s implementation of technology integration

According to the analytical results, three factors affected faculty’ use of technology in their classes (online or hybrid mode): a) challenge in fostering social interaction in class, b) appropriateness of the integration regarding subject contents, and c) perceived student cognitive load in online context.

a) Challenge in fostering social interaction in class:

I’ve looked for ways to make things more interactive, but it’s very difficult, but I would agree that there’s obviously a much different energy when you’re in a room full of people rather than talking to your computer. I’ve tried programming the computer to laugh at my stupid jokes, to no success.You know, at least at least, I get a sympathy, chuckle in a room full of students, but so that part is difficult.” ~ Focus group session, April 12, 2021

b) Appropriateness of the integration regarding subject contents:

The Excel portion of it. The whole first module is done through a simulator online, so there’s not a whole lot of lecture component to it other than working on exercises. And so that could almost be, you know, like the first few weeks online and then go into person when we get into the advanced topics in the middle of the semester.” ~ Interview with faculty, April 7, 2021

Regarding the use of HyFlex, “Like I said, having them remote did work better like yesterday’s last night session, where I could throw into breakout rooms was great. It worked better than if we would have been in person. So I could see, you know, (the) kind of switching back and forth between them depending on what we’re doing week to week and what not.” ~ Interview with faculty, April 7, 2021

Note: HyFlex is a course design model that presents the components of hybrid learning in a flexible course structure that gives students the option of attending sessions in the classroom, participating online, or doing both (Northern Illinois University, n.d.). 

c) Perceived student cognitive load in online context:

“I found it very interesting that in my previous class I had a lot of reading assignments, so I’m like, previous (one) is an in-person (class) where they had to do a lot of online reading and so there was still a reading component and when we became on line I didn’t add anymore reading. I actually took some of it away … Let’s just mix that until we’re back in person, but I never got complaints about the amount of reading when we were in person. Once we became online, (it was) like ‘there’s so much reading.’ And I’m thinking there’s less. There’s literally less reading, but I I think one of the things that could be happening is they have a mental block not just for being online, but also for the content, because (if) they show up to my lecture, (as) if they’re completely terrified of everything.” ~ Focus group session, April 12, 2021

3) Faculty’s perspectives on developing knowledge of technology integration or multimodal teaching

Based on the analytical results, faculty gradually developed capacity to integrate technology, pedagogy, and self-reflective teaching as indicated by the following examples:

“You know there are errors going on or the sound wasn’t going through, and so … I always had to show up significantly early, so it took …an extra hour every session just to make sure I’m there early.” ~ Interview with faculty, April 7, 2021

“There’s been more hand holding in this environment and more people that just need more. You know, more detailed explanations of things. And I’m not saying it’s bad or that they’re doing a poor job or it’s wrong, or whatever. It’s just, that’s kind of what I’ve experienced.” ~ Focus group session, April 12, 2021

“Because we’re teaching them about communication skills, we were putting them in rooms and saying do these things while you’re there and you’re going to come back and you’re going to talk about the conversations that you had, so we forced them to keep their cameras on essentially and talk to people, and they were like, ‘whoa, I didn’t realize how much I missed talking to people’… using breakout rooms as an opportunity for people to connect if they want to. If they don’t, that’s fine, but I think that’s probably the biggest success that came out of this unexpectedly.” ~ Focus group session, April 12, 2021

“That’s a huge advantage just even being able to bring people together to meet different guest speakers from all over the world easily.” ~ Focus group session, April 12, 2021

“The other big thing for me is that the course itself is just so much more like hyper organized because it has to be.” ~ Focus group session, April 12, 2021

“I really appreciate how well the class was organized and that came out of the, oh well, SOIREE …and frankly, even just something as simple as the course calendar …But the checklist I did not know that was on D2L. And I remember going through SOIREE, I think it was _____ (EDLI team member) …you talked about using this checklist that I, at the time, I thought, well, yeah, I’ll try that. And if it’s gonna make a big difference, huge,  huge difference.” ~ Focus group session, April 12, 2021

“This whole sense of being better organized is something I’m going to carry well into the post COVID world and the students really have reacted well to that.” ~ Focus group session, April 12, 2021

“If I have a student who is out for whatever reason like was ill or something like just knowing that I have a nice online backup, just like sending them all the links of past recordings. It makes me feel a little bit better about the future that I’ll be able to maybe help students who might not like (learning) online and only (like learning) in person.” ~ Focus group session, April 12, 2021

Be consistent. But what does that mean? My class is routine as it can get Monday, Sunday, Monday, Sunday, Monday, Sunday, Monday, Sunday. So what’s my schedule? Monday, Sunday? Reading due every now occasionally if there’s an exam, it might both be due on Sunday instead of Monday. But students said they always knew that if it was Sunday, there was something due tonight. There’s something due tomorrow. And then they get into the routine. I think the routine is really (the) key.” ~ Interview with faculty, May 5, 2021

“I am thrilled to see this substantive, academic account of the approach we are taking with the EDLI initiative. I appreciate in particular the references to the theory that animates our work.” 

Feedback from an EDLI stakeholder

Future plans: We will continue to guide the evaluation of the EDLI projects and activities based on the EDLI program evaluation plan. Meanwhile, we will also make adjustments to the plan or the process as needed, should significant changes to the EDLI program occur. 

Publication: Fu, J., Schopieray, S., Thomas, S., Van Hof, J., Wellman, S., & Kirby, C. (2021, November). Program Evaluation of the Enhanced Digital Learning Initiative (EDLI): Process and Takeaways. Paper presented at the annual conference of the American Evaluation Association.

Author/Personnel: by Jun Fu; Scott Schopieray, Stephen Thomas, Jeremy Van Hof, Sarah Wellman, Caitlin Kirby

References:

  • Dye, J. F., Schatz, I. M., Rosenberg, B. A., & Coleman, S. T. (2000). Constant comparison method: A kaleidoscope of data. The Qualitative Report, 4(1/2). http://www.nova.edu/ssss/QR/QR4-1/dye.html
  • Johnson, N., Veletsianos, G., & Seaman, J. (2020). US Faculty and Administrators’ Experiences and Approaches in the Early Weeks of the COVID-19 Pandemic. Online Learning, 24(2), 6-21.

Jun Fu

Dr. Jun Fu joined the EDLI Team from Oklahoma State University, where she earned a PhD in Educational Psychology and served as an adjunct faculty member at the School of Educational Foundations, Leadership and Aviation. Dr. Fu specializes in program evaluation, research and assessment for the EDLI team.