Are 3D printed models acceptable in assessment?
Funding information: UCLMS; UCL Changemaker
Abstract
Background
Three-dimensional (3D) printed models are increasingly used in undergraduate anatomy teaching. However, their role and value in anatomy assessment remains under consideration. The aim of this study was to evaluate student and educator perspectives on acceptability of using novel 3D printed heart models for assessment.
Methods
We used printed 3D models of the heart for first-year medical students, in small group teaching, formative assessment and revision at home. We adopted a mixed methods approach involving questionnaires, then focus groups to collect student and educator views. We used QSR Nvivo to manage thematic analysis of responses, carried out by student and educators, respectively.
Findings
Overall, students 89% (n = 75/84) and educators 91% (n = 10/11) found the assessment acceptable. Thematic analysis of focus groups (n = 4 students, n = 5 educators) identified five key perceptions shared across student and educator groups: 3D models are the future, realism is valued, models appear feasible, consistent and provide a potential for a range of applications in assessment.
Discussion
There was agreement between educators and students that the use of 3D heart models was acceptable. Key recognised benefits include accessibility and consistency across settings, made more relevant in the current COVID-19 pandemic. We recommend integration of 3D models into teaching and assessment for educational alignment and careful selection of anatomy to model. Further research is required to explore the use of models in summative assessments.
1 BACKGROUND
Three-dimensional (3D) printed models are gaining momentum in undergraduate anatomy education.1 They have been successfully incorporated into anatomy teaching, with the option of allowing students to take the models home.2 Proponents of the use of 3D model versus preserved anatomical specimens in teaching have cited key advantages, including lack of legal regulations, control over availability and integration into multimodal teaching for improved educational outcomes.2, 3
These advantages extend further to their use in anatomy assessment. As there are no legal restrictions on transport and use, this facilitates assessments taking place in a variety of hospital, university or home settings. The ease of access to readily available assessment tools is made more relevant in the context of the current COVID-19 pandemic. Additionally, 3D models improve assessment reliability as all students are assessed with the identical model. Candidates can directly handle and annotate the model as part of the assessment.
Access to readily available assessment tools is made more relevant in the context of the current COVID-19 pandemic.
Crucially, 3D models are anatomically representative of normal human anatomy, since high-resolution models are derived from computed tomography (CT) and magnetic resonance imaging (MRI)4 of real people with normal findings. Previous studies have demonstrated the validity of 3D models in teaching anatomy.5 As defined by the Ottawa criteria for assessment, validity here means “the results of an assessment are appropriate for a particular purpose as demonstrated by a coherent body of evidence.”6 Given an aim of assessment is assessing student application of knowledge to identify realistic anatomical features, 3D models derived from normal scans represent a valid tool for that purpose.
While there is a growing use of 3D models in teaching anatomy,7, 8 studies examining the acceptability and feasibility of assessment using 3D models9 remain limited. The Ottawa criteria for good assessment6 defines acceptability where “stakeholders (including but not limited to faculty and students) find the assessment process and results to be credible” and feasibility, a key consideration inherently linked to this, where “the assessment is practical, realistic, and sensible, given the circumstances and context.”
2 CONTEXT
At University College London Medical School (UCLMS), first-year medical students are assessed in an Objective Clinical and Practical Examination (OCaPE), a format similar to an Objective Structured Clinical Examination (OSCE). OCaPE assess integration of applied anatomical knowledge and feasibility is a particular concern. We considered 3D models as an alternative method of assessment, in part due to legal limitations on human tissues, which make their use in OCaPEs at multiple sites difficult. We were also concerned about the logistical challenges of implementing OCaPEs with human tissues within the timescales of the assessment cycle. Hence, feasibility is a key area of interest. Acceptability is also relevant, as we were concerned a change from assessment with human tissue to 3D models would not be acceptable to students and educators. These components are essential to a “good” assessment process.6
At UCLMS, we produced 3D printed heart models from a database of STereoLithography file (STL) files, which describe surface geometry of a 3D object in computer-aided-design software for 3D printing. These files were obtained from a publicly available database of STL files derived from CT and MRI image files.10 This was achieved with support from the Anatomy Laboratory and Bartlett School of Architecture. We carefully selected and reviewed prototypes, consulted two members of the senior anatomy faculty to review anatomical features modelled. The finalised model is shown in Figure 1.
As there a body of literature on 3D heart models in teaching undergraduate heart anatomy, it is key to connect our study of assessment methods with existing methods of teaching.5 Alignment of teaching and assessment methods is shown to have important educational benefits.11 We chose a heart model to investigate the acceptability of a 3D model for assessment as there is a paucity of cadavers without signs of previous coronary surgery and difficulty visualising the posterior aspect of the heart in 3D during cadaveric dissections. The features included in the 3D model correlate with learning objectives in the UCLMS anatomy curriculum, including key vessels such as the left anterior descending artery, left circumflex artery and right coronary artery.
The model was integrated into small group teaching of cardiac anatomy for first-year medical students. The same model was used in a group viva-style formative assessment for all first-year students. The Anatomy and Assessment faculty both reviewed the assessment design, setting a format which required students to directly interact with the model to label anatomical features of the heart and answer relevant questions. Afterwards, we gave each student a model to take home for revision; however, due to the COVID-19 pandemic, summative assessments did not include practical assessment of anatomy.
The model was integrated into small group teaching of cardiac anatomy for first-year medical students.
Given the opportunities offered by 3D models in assessments, this study aims to evaluate the acceptability and feasibility of this novel assessment tool, from both student and educator perspectives.
3 METHODS
Aligning with the nature of an acceptability and feasibility study, we employ a pragmatic philosophy12 and a mixed methods approach. We used questionnaires to collect anonymous data conveniently, then semi-structured focus groups to develop issues that emerged. We conducted semi-structured interviews to strike a pragmatic balance between exploring participant perspectives while allowing a structure to confer reliability and validity.
Participants for the study were identified by their medical school year group. The entire cohort was taught using the 3D model and took the same formative assessment which included the 3D model. Two weeks after the assessment, following a lecture the whole year group were asked to voluntarily complete a live questionnaire. Six weeks later, a focus group was conducted via Microsoft Teams with those who expressed an interest in participating through the questionnaire. Similarly, we sent questionnaires to all educators involved in anatomy teaching or assessment using the 3D model and invited those who expressed interest to the focus group 4 months later. The development of the questionnaire was informed by Denscombe's The Good Research Guide,12 with educator questions quoting relevant technical terms based on the Ottawa criteria, while student questions contained non-technical language. Questionnaire design was an iterative process of review by the assessment faculty and student investigators to ensure wording and purpose were clear. An educator investigator wrote both questionnaires, and student investigators edited the student questionnaire to clarify terminology and minimise medical education jargon. The finalised questionnaires are shown in Table 1. We reviewed response to ensure they aligned with the questions asked.
| Student Questionnaire (MentiMeter – instant responses) | Educator Questionnaire (Typeform – via email) |
|---|---|
|
1. Did you attend an anatomy lab teaching session involving scaled down 3D printed model hearts? - Yes/No |
1. What is your role in anatomy teaching? - Academic - Senior honorary demonstrator - Near-peer demonstrator - CT2 surgery demonstrator - Honorary demonstrator - Senior near-peer demonstrator - Other - No role |
|
2. Did you attend a formative anatomy practical assessment and feedback session? - Yes/No |
2. What was your involvement in formative anatomy practical assessment? - Assessment Design - Assessing Students - Feeding back to students - Other |
|
3. Was the 3D printed heart formative practical assessment acceptable to you? - Yes/No |
3. How would you rate the validity or coherence of the formative anatomy practical assessment involving the 3D printed heart? - Likert scale 1–5 (where 1 = Very Poor, 3 = Satisfactory, 5 = Excellent) |
|
4. Why did you think it was or wasn't acceptable? - Free text |
4. How would you rate the consistency or reproducibility of the formative anatomy practical assessment involving the 3D printed heart? - Likert scale 1–5 (where 1 = Very Poor, 3 = Satisfactory, 5 = Excellent) |
|
5. Has the 3D printed heart impacted your learning since then? - Yes/No |
5. How would you rate the equivalence of the formative anatomy practical assessment involving the 3D printed heart? - Likert scale 1–5 (where 1 = Very Poor, 3 = Satisfactory, 5 = Excellent) |
|
6. If Yes – Please describe how it has impacted your learning. If No – Why do you think it did not impact your learning? - Free text |
6. How would you rate the feasibility of the formative anatomy practical assessment involving the 3D printed heart? - Likert scale 1–5 (where 1 = Very Poor, 3 = Satisfactory, 5 = Excellent) |
|
7. What other uses could you see for 3D models? - Free text |
7. How would you rate the educational effect of the formative anatomy practical assessment involving the 3D printed heart? - Likert scale 1–5 (where 1 = Very Poor, 3 = Satisfactory, 5 = Excellent) |
|
8. Please rate how you feel about the following in relation to the use of 3D models in assessment: (Likert scale 1–5 where 1 = Strongly disagree. 5 = Strongly agree)
|
8. How would you rate the catalytic effect of the formative anatomy practical assessment involving the 3D printed heart? - Likert scale 1–5 (where 1 = Very Poor, 3 = Satisfactory, 5 = Excellent) |
|
9. Would you be interested in sharing your thoughts on 3D models in a short online focus group? If yes, please type your email: - Free text |
9. How would you rate the acceptability of the formative anatomy practical assessment involving the 3D printed heart? - Likert scale 1–5 (where 1 = Very Poor, 3 = Satisfactory, 5 = Excellent) |
|
10. Please give any other comments or feedback about the formative anatomy practical assessment using the 3D printed model of the heart - Free text |
10. Do you think it would be acceptable to have a similar station, using 3D models, as part of the summative assessments? - Yes/No |
|
11. Why did you give this answer? - Free text |
|
|
12. If you would consider participating in a short online focus group to explore the responses to the questionnaire, please include your email below: - Free text |
Focus group questions were developed guided by issues highlighted from respective group questionnaire responses.13 The time lapse between the assessment and focus groups was due to the COVID-19 pandemic; educators were less available as they rapidly shifted to online teaching. Student and educator focus groups were led by student investigators and educator investigators, respectively, to encourage open discussion and improve validity. There were two student investigators (SWLY and NM) leading the student focus group and thematic analysis. SWLY has been a previous participant in focus groups. Educator CH provided student investigators with informal training and guidance ahead of their focus group and analysis, including an overview of thematic analysis, and mentorship throughout the process. During the student focus groups, CH remained silent but available for support. Supervision was provided by AS (head of UCLMS assessment faculty), GM and CH who have extensive years of medical education experience and training at master's level and above.
The research was carried out within normal educational requirements and with full informed consent. During the pandemic, ethical approval was not required under UCL guidelines for the aforementioned reasons and as the study was considered by the authors as low risk with no potential harm to the participants. All focus groups were audio-recorded, anonymised and transcribed with written participant consent, as compliant with GDPR, and deleted after analysis. The participants also consented for the use of anonymous data in publications.
We thematically analysed interview transcripts14 with QSR NVivo and derived descriptive statistics for quantitative data. We generated initial codes from reading and re-reading the interview transcripts. Themes were iterated independently by respective investigators, then discussed and reviewed by all authors. Where agreement could not be reached on defining and naming themes between investigators SWLY, NM and CH, the senior investigator AS made the final decision. Finally, the educator investigator merged the overarching themes identified in both groups. We acknowledge the reflexivity of the investigators involved in thematic analysis. The student investigators are both members of the university Anatomy Society, hence are inherently interested in how anatomy is assessed, and they have found the concept of learning from 3D models appealing. The educators have a stake in the success of implementing 3D models as a convenient method of delivering formative OCaPEs such that this may be used in further assessment cycles; however, they would also like to ensure this process is credible to stakeholders and feasible for assessments in the future.
4 FINDINGS
We collected 84 student responses (response rate 25%) and 11 educator responses (response rate 31%) to the questionnaires. When given a binary choice on whether 3D models are an acceptable assessment, students 89% (n = 75/84) and educators 91% (n = 10/11) found the assessment acceptable. Four students and five educators took part in the student and educator focus groups, respectively.
We extracted several key themes from thematic analysis of the qualitative data from student and educator focus groups and questionnaire free-text responses, as shown in Figure 2.
We found 3D models were an overall acceptable compromise for assessment. Illustrative examples are displayed in Table 2. “S” refers to student participants, and “E” to educators, the number being allocated by their order of contribution to the relevant focus group.
| Overall acceptable compromise for assessment | |
|---|---|
| E3: “I definitely think it's acceptable, and I think that what it comes down to for me is the compromise that we have to make with this new world of not being able to assess everyone with a spot test” | S1: “I thought it was very acceptable, I really liked it, and for the assessment I was like yeah I agree.” |
| E2: “I think this is a pretty good compromise, it's about as good as it gets” | S3: “Not all hearts have arteries and veins easy to identify, so having a model made this more acceptable” |
This overall finding is supported by five main themes shared by educators and students, as well as student- and educator-only themes, respectively. The shared themes extracted from student and educator groups are presented by theme in Table 3.
| 3D Models are the future | ||
|---|---|---|
| Future use of 3D models | E2: “you know this imaging is now already going towards the 3D imaging in radiology … so when I saw that I really felt that this is the way forward” | S4: “I think it would be a nice addition, and I can see it being in an assessment in the future.” |
| Realism Valued | ||
|---|---|---|
| Cadaver preferred |
E5: “That's really the case for using cadaver material is that you get that human variation side of it.” E3: “us instructors are old fashioned enough to think that a spot test is the best way to assess this” |
S3: “people were maybe slightly worried about it not being the real thing, … well you know obviously a real organ would be a much better thing to use.” S2 “You kind of prefer … like want to do it in like a cadaver setting” S1: “that was quite useful actually just to see kind of what a regular heart looks like and then testing that, as well as obviously testing the cadaver.” |
| Unrealistic, idealised | E3: “I think there were some real issues around realism and colour and size” | S4: “I just think looking at the perfect example first and then looking at different real examples is probably the better way to go” |
| Too small | E4: “ideally I would go for life size as often as possible, but I agree with E1. that if it's not possible then small is better than nothing” | S2: “we are pointing to a vessel that's coming off a main one or something else, because it was quite small.” |
| Feasible | ||
|---|---|---|
| Creating Larger Model Requires More Resources | E1: “3D printer that could do multiple colours and larger objects, but that would be appropriate for large multi-coloured objects, but very few of them. For example, if we were going to do one heart per table … if we are doing one heart per student it's not cost effective to do anything else.” | S2: “like making them larger would just probably be more expensive and stuff. It's probably better to like you know save the money for maybe other similar projects later on” |
| Logistical Benefits | E4: “can be used anywhere” | S1: “I liked the fact it was light, I liked the fact it was quite small - there's pros and cons, because if you make it bigger it's kind of harder to carry around or like if you want to use it and take it with you.” |
| Application in Assessment | ||
|---|---|---|
| Allows assessment broader range |
E1: “It would allow to test many different aspects of anatomy in 3D, which would make it more inclusive what can be tested” E4: “we could maybe produce a different summative model – even if it's the same structure … you use that one year – and maybe two years later you produce a different such model from a different CT scan, it would be very very different” |
S1: “you could use the model … (to) express like differences between the right and the left and be like well you can see that the left one is bigger, the left ventricle is bigger, and you can ask questions from like that” S3: “taking the same approach to maybe a different organ” |
| Consistent | ||
|---|---|---|
| Consistency | E2: “very clear that everybody rates using the same model everywhere – that's the consistency” | S1: “because people's hearts are different, and this is like obviously more consistent.” |
|
Equivalence |
E4: “I think that the strength … to me like one of the reasons that we were trying this whole plan was to get a way to have equivalence for assessing across OCAPE sites.” | S2: “If you are trying to make it fair to everyone because they are different sites … that model probably would be a good alternative, just to make it slightly more fair.” |
Comments on the difficulty of the assessment method were varied from students while no theme on this emerged from educators. Some expressed it was too easy; S2: “you can't make it any harder to identify stuff on a model.” Other students maintained it was of a similar difficulty to other formats; S1: “if you'd used it on the main heart of the cadaver or in a textbook it would also be in the same level of difficulty.”
Educators highlighted that the 3D model aligned assessment with teaching, as the same tool used in teaching was also used to assess knowledge, reinforcing key learning objectives. E2, “everyone did very well because … they had seen it, they had learned on it.”
The 3D model aligned assessment with teaching, as the same tool used in teaching was also used to assess knowledge.
5 DISCUSSION
Both quantitative and qualitative data demonstrate that the use of 3D models in formative assessment is acceptable overall, to students and educators at UCLMS. Notably, there was much agreement in perceptions of the value of these models. Both student and educator groups recognised the benefits of accessibility, consistency and equivalence in using the 3D model for assessment. They shared concerns over the trade-off between resources and fidelity, when speaking of modifying design and printing the models. Their correlation of perceived benefits and concerns despite their respective positions in the subject is striking. However, on the following areas, their nuances in perspectives are explored.
Both student and educator groups recognised the benefits of accessibility, consistency and equivalence.
During the focus groups, educators focused on alignment and credited the 3D model with allowing constructive alignment of teaching activities and assessment strategies. Conversely, students spoke separately about learning and assessment. They commented that the model was a useful revision tool used alongside cadaveric dissection, and some felt the model served as a motivator for further learning. Many educators felt assessing with the model could result in narrowing of student knowledge of anatomy to one “normal” without account of human variations. However, this concern appears unfounded as students view the model as an “additional” assessment tool they can access any time or place. Hence, the model acts in conjunction with other multimodal assessment and learning strategies.3 In terms of suggestions for improvement, educators focused on design of the current model and greater integration of 3D models into the anatomy course while students explored other possible organs to model such as the lungs. Both groups were interested in the applications of the model to anatomy assessment; however, their viewpoints differ subtly. Students focused on its use in viva-style questions directly applicable to themselves, such as using the model to ask multi-step questions on the heart. Educators discussed the long-term variations across the assessment cycle and the long-term limitations.
The model acts in conjunction with other multimodal assessment and learning strategies.
There are a variety of factors which aided implementation of assessment with a 3D model in this context. For instance, we selected anatomy that may be difficult to view in situ in a cadaver, consulted anatomy experts and stakeholders, as well as facilitated the use in OCaPE assessment. This format examines integrated anatomy and clinical knowledge, in contrast to a traditional cadaveric spotter exam.
This study is limited by the low questionnaire response rates, involvement of only one first-year cohort albeit in a large medical school, and evaluation of 3D models for only formative assessment. Due to the COVID-19 pandemic, the availability of students and educators was reduced, impacting response rates and availability for focus groups. It is possible that students and educators may not recall the details of the assessment as optimally for depth of responses. However, the 6-week delay for the students was important, as pragmatically, it would have been difficult to recruit students the weeks directly following the assessment. Also, students were given the models to take home and revise, giving a reminder of the assessment. Although not the focus of this study, this allowed exploration of whether assessment motivated further anatomy revision, which was only possible with adequate time for reflection.
Unfortunately, due to the pandemic, follow-up summative examinations were cancelled. Future work will need to explore results following summative examinations. Formative assessments are designed to monitor learning and provide feedback whereas summative assessments evaluate the learning of students.15 For example, the comments relating to the assessment being too “easy” are unlikely to be a criticism from students in the higher stakes summative assessment, as opposed to in formative assessment where feedback is provided to drive further learning.
Despite these limitations, we can still draw valuable conclusions and implications for our future 3D modelling work. Overall, students and educators found the model an acceptable assessment tool, based on benefits gleaned from its consistency, equivalence and logistical accessibility. 3D models also open the possibility of online assessments in any setting. However, the model is limited as a smaller, unrealistic representation, as supported by previous literature.2, 9 Both groups expressed a preference for assessment using “real” anatomy. We recommend careful selection of anatomy and consultation of stakeholders when designing the model, particularly in decision-making around feasibility (including scale and cost) to best align with learning outcomes.
Overall, students and educators found the model an acceptable assessment tool.
We recommend careful selection of anatomy and consultation of stakeholders when designing the model.
ACKNOWLEDGEMENTS
We thank the UCL Department of Anatomy, particularly Ryan Felice and Sandra Martelli, and the Bartlett School of Architecture for their support with producing the 3D printed model. We would also like to thank Brighton and Sussex Medical School for their advice regarding the 3D model and the UCL ChangeMakers scheme for this research.
We acknowledge funding from UCL Changemaker Grant of £850 and UCLMS (models).
CONFLICT OF INTEREST
The authors have no conflict of interest to disclose.
ETHICS STATEMENT
Under the UCL guidance for research and ethical approval during the COVID-19 pandemic, this study, which fell within normal educational requirements, was categorised as low risk and did not require ethical approval. Informed consent was obtained from all participants.
