Assessment Drives Transformation Across University Departments

You might call them “The Academic Exercises.”

Almost 500 years ago, St. Ignatius Loyola, founder of the Society of Jesus, wrote a meditative guidebook known as “The Spiritual Exercises.”

At a basic level, the exercises serve as a compass of sorts for those looking to reflect on what works and doesn’t work on their spiritual journey. In the early stages, one key step is the recognition of shortcomings as part of the human experience and the realization that such recognition can pave the path for transformation.

At The University of Scranton, a Jesuit university, professors are familiar with that concept, as it can apply not only to religious journeys but academic journeys as well. In the latter case, the steps taken on the journey are part of a broad concept known as academic assessments, and the findings can be as altering as spiritual assessments.

Reflections transform Special Jesuit Liberal Arts Honors Program

To hear Rebecca Haggerty, assistant dean for assessment and programs for the University’s College of Arts and Sciences, tell it, assessment not only can be transformational for faculty but reveal transformations in individual students.

Haggerty said she and her husband Daniel Haggerty, Ph.D., director the University’s Special Jesuit Liberal Arts Honors Program (SJLA), have long known SJLA, an esteemed honors program for the University’s top-tier students, was stellar.

“Generally speaking, everyone at Scranton talks about how amazing SJLA is,” she said. “We saw that firsthand, but we wanted to know why.”

Thus began the process of assessing one of the University’s signature honors programs not only from a hard-data standpoint, collecting statistical information, such as grade point averages and classes taken, but through a softer lens as well, the lens of personal reflection.

Every two years SJLA students and faculty embark on a mission-based trip to Spain and Italy, a journey that puts them up close and personal with the spiritual journey of St. Ignatius, Haggerty said. Upon return, the students were asked to write an essay about what the experience had meant for them, and that’s when major revelations struck.

“Everything they said was amazing, and they realized how important reflection actually is,” Haggerty said.

The mutual realizations led the Haggertys to do a different type of essay-based assessment after each trip. The lessons were so revelatory it was decided students needed even more time to devote to reflection.

A companion course was then developed for SJLA students to take each fall after their summer mission-based trip. The course, “The Loyola Experience: An Ignatian Pilgrimage,” focuses heavily on reflections, Dr. Daniel Haggerty said, noting that students became increasingly aware of their personal journeys and how intricately connected they are to their Jesuit education.

The Haggertys discussed their reflection-based approach at a recent Drexel University Assessment Conference, making a presentation that surprised many in attendance as it took the position that transformational experiences are indeed quantifiable.

A survey of SJLA alumni from every class since 1980 further illustrated their case.

“We had an astonishing 40 percent response, receiving 1,240 comments composed of 45,000 words,” said Dr. Daniel Haggerty. “When asked what beneficial career skills SJLA helped develop, 94 percent of respondents said writing; 94 percent said critical thinking; and 91 percent said public speaking. We learned that 70 percent graduated with double or triple majors with 81 percent going on to earn Ph.D.s, M.D.s, J.D.s, and M.B.A.s.”

The business side of writing

Assessments done at the University’s Kania School of Management (KSOM) between 2005 and 2010 led faculty there to take several actions to improve student performance in communication, particularly writing skills.

Now, in essence, the business track has become increasingly communication-heavy. Oral-presentation components were added to two existing courses, a new written and oral communication elective was added to the accounting track, and 12 new writing-intensive courses were designated within the general KSOM curriculum. Faculty members in the new courses were encouraged to focus heavily on communication outcomes as well. Faculty also were encouraged to adopt a zero-tolerance writing-error policy for freshman seminar courses.

As a result, student outcomes in oral and written communication are continually improving, and individual faculty members continue to take steps to increase success, whether by providing more writing feedback, imposing explicit penalties for spelling and grammatical errors, or providing opportunities for writing revision post-feedback.

Assessing the physics of connections

In the University’s Physics-Engineering Department, Professor Nicholas Truncale and colleagues used assessments to identify several ways to increase retention in an admittedly difficult major.

“This all started four years ago when we did a program review for all majors in our department – physics, biophysics, computer and electrical engineering, and engineering management,” Truncale said.

The faculty engaged in a group effort, he said, to look at four semesters’ worth of data related to incoming students’ math SAT scores, high-school GPAs and scores on the math-placement exam.

Existing math requirements illustrated a case of how what was once thought a service to students can become a disservice. Precalculus had been required of department majors taking Physics I in the freshman year, but relaxed requirements eventually allowed them to take both courses at the same time, Truncale said.

After reviewing data, it seemed that enforcing the precalculus prerequisites would not be a hardship to students, but would rather achieve better results.

The solution implemented was to create a “trailing sequence” through which students, depending on their math placement scores, could take precalculus in fall, then calculus along with Physics I in the spring.

“This helped a lot,” he said, but it also contributed to a slightly different problem: disconnectedness among what were now two tracks of students.

“We didn’t want to have this big split,” Truncale said, who noticed along with other professors that students were not conversing as much with one another in classes or staying in physics labs later in the evening, which had been the norm with students in previous cohorts.

Enter a second solution to a new concern. When the general education program added the Eloquentia Perfecta (EP) initiative, designed to prepare students to speak, write and communicate effectively in varied modes and media, departments were mandated to choose one of three approaches.

Truncale and his colleagues chose to create a new course highlighting EP requirements in addition to certain departmental objectives. Created was a Foundations of Physics and Engineering course that focuses on oral communication and digital technology outcomes. But the course had other imbedded elements, such as group projects and team competitions, which were intended to close the connection gap among students in different math sequences.

“I noticed a difference immediately,” Truncale said. “The students were now hanging out together, studying together.”

The results of these changes, in turn, helped increase retention, as students looked forward to taking on fun physics and engineering projects together, such as creating a self-navigating robot. They were now able to see what life in the department would be like “after they got past the hard math,” Truncale said.

“This year we had 32 freshmen, and we’ve only lost two or three,” Truncale said.

Another tough litmus test for an already successful program

In the University’s Nursing Department, assessments also have been proving pivotal to success in an already stellar program, and one in which a licensing exam is the key to a future career.

Faculty discovered that early intervention is key to helping students achieve success not only at Scranton but on “a very complicated, very rigorous national exam,” said Margarete Zalon, Ph.D., professor of nursing.

“The University is always charged with doing assessments,” Dr. Zalon said, figuring out “what works, what might work better, what can we fix?”

But in a clinical program such as nursing, she said, the most crucial question to ask would be: Are our graduates doing well in the licensure exam?

“The way we define it is if they can pass the exam on their first attempt,” Dr. Zalon said. “And, although Scranton’s first-time pass rate averages 92 percent for the past five years, compared to the national average of 85 percent for the same time period, there are still ways to look to data to lead improvement. Not passing the exam on the first attempt is a disappointment to the new graduate, but has serious financial implications with delay in getting into the job market and student loans coming due.”

“One of the things we do know is that while we accept students to the program who we believe are going to succeed, some students will have more difficulty than others,” Dr. Zalon said. “We wanted to ask: Can we identify students in need of support services early on?”

Dr. Zalon and fellow nursing faculty member Mary Jane K. DiMattio, Ph.D., found in their assessments that grades in certain early courses, such as first-year science courses, are predictors of future exam success. Students who struggle in these courses can be identified for early support and steered toward it, Dr. Zalon said.

“When we go to a student and say, ‘You need to work on this,’ we’re not going out of gut feeling, but we’re basing it on data,” Dr. Zalon said explaining that the use of data yields credibility to the dialog the faculty members have with students.

Students are often advised to prioritize activities or revise study strategies. The latter can be an eye-opener, Dr. Zalon said; handling test questions based on ability to analyze a situation rather than recall information is a new skill.

“Sometimes students just have to learn how to take this type of test,” Zalon said.

And, throughout The University of Scranton, academics are using “Assessment Exercises” to learn how to be more effective teachers.