Serious Games Design, Validation & Accreditation Framework
18 March 2020
Dr Bina Rai
Senior Lecturer, Science, Math & Technology (SMT) Cluster
Program Director, SUTD-CGH Healthcare Collaboration &
Senior Fellow, SUTD Academy
Singapore University of Technology & Design
Serious Games Association
Why is the SG-DiVA Program necessary?
Let us start by defining the terms commonly used in Serious Games. We would like to take the definition from Ifenthaler et el, serious games are digital games and simulation tools that are created for non-entertainment use, but with the primary purpose to improve skills and performance of play-learners through training and instruction.
Game-based Learning (GBL) refers to both the innovative learning approach derived from the use of computer games that possess educational value and other software applications that use games for learning and education purposes (e.g. learning support; teaching enhancement; assessment and evaluation of learners etc.). Serious games are one of such computer software.
Game-based Assessment (GBA) are assessments built as games to assess learners’ skills in a quick and engaging experience. It is often applied as a game-based learning pedagogical approach to demonstrate if the learners have indeed achieved the proposed learning goals.
Gamification is not a game. It borrows from the concept of game mechanics to motivate people to continue certain behaviours – such as posting photos on Facebook, encouraging the sales force to work harder, etc., through point systems, badges, or monetary awards. Gamification can be used in conjunction with digital games but it is not a new type of digital game in and of itself.
As computer gaming becomes a digital culture deeply rooted amongst the new generation of learners, many educational researchers and practitioners agree that it is now appropriate to exploit gaming technologies in order to create engaging interactive learning content to motivate learners to learn through game-playing using the game-based learning and subsequently lead us to more sophisticated level of game-based assessment.
Currently, the adoption rate of game-based learning (GBL) and game-based assessment (GBA) is still low. One of the barriers to the adoption of GBL & GBA is the extremely steep learning curve required to create serious games. Most of the digital games available in the market are designed for entertainment purposes.
There are several ways which serious games are being created at this moment.
- Using In-House Resources: This has led some subject-matter-experts experts to create serious games through either in-house development team using open source or royalty-free game engines and “modding” (or modifying) commercial-off-the-shelf games or training their in-house development team to use commercial game development engines. Many of these tools and technology platforms for producing serious games are readily available but most of these tools require substantial technical knowledge in games development which hinders non-technical domain experts from adopting games-based learning. The process for in-house development teams to get to become as efficient as commercial developers would depend on the base qualification of the in-house development team in terms of game development.
- Using Commercial Vendors: The second option is for subject-matter-experts to work with game development vendors to create the game. This option would require substantial funding as commercial vendors for game development are rather expensive due to high manpower cost. There may still exist the concern with the level of understanding and experience of the serious game vendors. Entertainment or casual game developers may not have the necessary skill sets that is needed for serious game development and thus the process may be still a painful partnership between the subject matter experts and the commercial game developer, which may result in substantial delay and cost increment.
- Using Game Authoring Tools: The third option was for domain experts to make use of high-level game authoring tools for non-technical individuals (i.e. teachers, clinicians, trainers) to create custom serious games. This option is still a risky move as it depends on the sophistication of the game authoring tool. The simpler game authoring tool would still require the subject-matter-experts to develop their own game development codes while the sophisticated game authoring tools come with a high price.
- Using a Hybrid Serious Game Development Approach: In order to balance the high cost of commercial game development vendors and not so sophisticated game authoring tool, there is a possibility of getting the commercial game development vendors to create the base-scenario evidence-based serious game while the in-house development team uses a fairly sophisticated game authoring tool kit to create subsequent game scenarios to make a complete serious game. This will allow the cost of game development in partnership with a commercial game development vendor to be kept at a relatively low cost yet benefiting from the expertise and experience of the vendor.
That said, all four options would still require the subject-matter-experts, the instructional designer and the serious game developer to have a fundamental, shared and common knowledge in serious game design and development. This is also a form of standardising the common terms used in such projects, setting expectations for each serious game project and establishing quality assurance for the games.
On this basis, Dr Bina Rai and Ivan Boo has developed the Serious Games Design, Validation & Accreditation (SG-DiVA) Framework. This framework is an adaption from previous work done by Sarah Verschueren and her team from Mindbyte, M.E.W Dankbaar and her team from the Dutch Society of Simulation Health and Konstantin Mitgutsch and his team from MIT Game Lab. The SG-DiVA framework is specifically targeted at evidenced-based and commercial-ready serious games.
The Serious Games Design, Validation & Accreditation Framework comprises of 3 distinct stages.
- Serious Games Domain Expertise (for Main Role Players)
- Serious Games Design, Development and Validation
- Serious Games Accreditation (for Commercial Ready Games)
Serious Games Design, Validation and Accreditation (SG-DiVA) Framework
For each serious game design and development, there must be a process of validation before implementation. This is where we adopted the Mindbytes Evidence-based Serious Games Validation Model for developing Serious Games for Health by Verschueren et al. In this serious game’s design, development and validation framework, there are 6 stages as reflected in Figure 1.
- Scientific Foundation
- Design Foundation
Stage 1: Scientific Foundations
Sound scientific foundations for the serious games should be established at the earliest stage of development. This will ensure that the final product is relevant, theoretically driven, and evidence based, in line with governing research methodological approaches. Although most serious games teams tend to initiate the development process with a specific idea for an intervention in mind, the overarching objective of this stage is to assess at least conceptually and theoretically, on the basis of objective criteria, whether there is indeed a relevant unmet need for a clearly defined target audience who can be addressed with an intervention. This stage typically comprises a top line review of the available literature on the target audience, status and impact, available intervention modalities and relevant outcomes, psychosocial aspects (if any).
At this stage, the team should already establish the purpose of the game. The purpose may be further refined into several sub purposes. The process of re-wording, adjustment and even re-purposing the serious game is a continuous iterative process. The following templates are developed for each team to use as project management tools:
Stage 2: Design Foundations
Serious game teams can draw from a wide range of game mechanics, design, and technological features to construct serious games. If the serious games are to achieve the intended outcomes, the choice of these game mechanics, design, and technological features should be guided by the scientific foundations established in stage 1. At this stage, it is imperative to translate the theoretical basis into relevant, implementable game design elements. It is important to have a set of game design guidelines establish to help the different role players in the team. Below, we have provided a set of 70 questions to guide the team.
Stage 3: Game Development
Once stage 2 has been completed, the serious game team should have sufficient scientifically grounded input to guide the practical development of the game. Various approaches can be used depending on the complexity, the resources, and software and technological skills, but overall, the process comprises the selection and development of the (1) game genre, (2) game rules, (3) content, and (4) visuals and user interface. This stage occurs in an iterative, participatory manner, involving the subject-matter-experts, instructional designer, the game developer and target users to informally test and refine the tool along the way.
Stage 4: Game Validation
The game has now been developed, informally tested, and refined with users, and it should be ready for validation. Once the trial sites and investigators have been chosen, ethical committee or other approvals have been granted, and users have been recruited for the evaluation study, developers can commence the evaluation, analysis, and assessment of whether the tool successfully achieves the intended outcomes.
If we regard serious games as purpose-based game systems, the driving force that functions as the pivotal influence over the elements of the game design should be the purpose of the game. Thus, the purpose should be reflected in all the elements that support the game system: content, the fiction & narrative, the mechanics, the aesthetics & graphics, the framing and the emotion, motivation, and memory. The relation among these seven core components impacts the coherence and cohesiveness of the formal conceptual design of the holistic game system. The pivotal function of the serious games system is not only to decompose each of the design elements, but also to examine how they holistically relate to each other and to the game’s purpose. One can argue that in games the whole is greater than the sum of its parts – but if the system is not cohesive, the whole can also be in a conflict.
We noticed that the framework is good as a top down overview for any serious games project, but it still lacks the specific guidelines for the subject matter experts, the instructional designers and the game developers when they are involved in the design and development phases. Subsequently when the game reaches the validation stage, where the domain experts with the support of the relevant research experts will oversee the evaluation studies and analysis of collected data, there is the same need for these guidelines.
Thus, we adapted the Serious Game Design Assessment Framework by Mitgutsch et al, which we identified the six (06) essential components, to help us in developing the specific guidelines. Dr Bina Rai and Ivan Boo added the 7th factor – Emotion, Motivation & Memory with importance placed on the motivation dimension when developing serious games. (See Figure 2)
With seven contributing factors directly affecting the design and development of serious games, we used the same set of guidelines for the design stage to review each serious game project. The set of guidelines would it clearer for subject matter experts, instructional designers and game developers to guide them in each of their role.
The list of 70 questions in total was created for the 7 contributing factors. The full list is provided below:
Establish clear intentions
|1. Is there a clearly defined problem/challenge statement?|
|2. Are there intended outcome objective(s) established for the game?|
|3. Are the intended outcome objective(s) established for each stage/each level of the game?|
|4. Are the intended outcome objective(s) relevant to the problem/challenge statement?|
|5. Is there a hypothesis of how a game might achieve the intended outcome objective(s)?|
|6. Are biological (human computer interactions) factors considered when establishing the intended outcome objective(s)?|
|7. Are psychological factors considered when establishing the intended outcome objective(s)?|
|8. Are social (human behaviour) factors considered when establishing the intended outcome objective(s)?|
|9. Are the intended outcome objective(s) visible to users?|
|10. Did the game achieve the intended outcome objectives?|
Reflect accurate content
|11. Is there a storyline/theme for the game?|
|12. Is the content organized around key concepts and principles that are fundamental to understanding the topic structure?|
|13. Is the content divided into stages/chapters following the topic structure?|
|14. Is the time duration of each stage/chapter distributed evenly?|
|15. Is the content accurate (realistic) of the topic?|
|16. Is the content flow consistent throughout the game?|
|17. Are 90% of the content coming from the domain experts?|
|18. Are all the intended outcome objective(s) addressed in the content?|
|19. Are the game interactions correlated to the intended outcome objective(s) at each stage?|
|20. Are there feedback/rewards/penalties at each stage?|
1.3 Fiction & Narration
Narrative in line with purpose
|21. Is there a narrative/story used for the game play?|
|22. Is the narrative consistent with the intended outcome objectives established for each stage of the game play?|
|23. Is the context used appropriate for the game play?|
|24. Is the narrative consistent throughout the game play?|
|25. Is there a back story to explain the game prior to the start of the game play?|
|26. Is there a climax in each stage of the game play?|
|27. Is there a resolution in each stage of the game play?|
|28. Is the user able to relate to the main character in the narrative?|
|29. Is narrative discourse built into the game play?|
|30. Is the narrative able to encourage self-reflection?|
Application of game mechanics
|31. Are the game mechanics aligned to the narrative and the intended outcome objective(s)?|
|32. Are the rules (in-game goals, reward system, penalties, scoring system, etc) applied in the game mechanics consistent throughout the game play?|
|33. Does the game apply relative weightage of scores for each intended outcome determinants in the game mechanics?|
|35. Are special equipment or facilitator intervention needed in the game play?|
|36. Is data collected from the special equipment or facilitator?|
|37. Is the user’s performance tracked?|
|38. Is live feedback needed during game play?|
|39. Does the game produce the same results on different platforms?|
|40. Is a user trial planned to validate the game play?|
Appropriate use of graphics
|41. Is the game environment aligned to the narrative and theme/storyline?|
|42. Can the user relate with the avatar used in the game?|
|43. Does each non-playing character has a role in the game?|
|44. Does the setting in the environment provide sufficient realism for the game play?|
|45. Are the patterns, routines and motifs consistent throughout the game play?|
|46. Are the avatars and assets used in the environment consistent?|
|47. Do the avatars and assets take into consideration cultural/country differences?|
|48. Do the audio sounds enhance the game play?|
|49. Do the instructions (wordings) enhance the game play?|
|50. Are the instructions (wordings) well balanced in the game play?|
|51. Is the target user defined clearly (age, gender, education, etc)?|
|52. Is the target user’s game literacy established?|
|53. Is the target user able to understand the expectations at each game stage?|
|54. Is the target user’s prior knowledge activated and connected to new learning?|
|55. Is the difficulty levels balanced in relation to the target user’s ability?|
|56. Is the target user having trouble using the controls?|
|57. Is the target user having trouble understanding the interface & instructions?|
|58. Is there possible physical or mental limitations that pose restriction for the target user?|
|59. Is the assessment of the target user clearly defined in each stage of the game play?|
|60. Is the target user able to relate to the assessment feedback?|
1.7 Emotion, Motivation & Memory
Consideration for repeat usage
|61. Is the game considered fun by the target user?|
|62. Does the game leverage on target user’s past experience?|
|63. Is the game able to reflect target user’s competency level in the game?|
|64. Does the game allow autonomy for the target user in the game?|
|65. Is the target user able to relate to the game?|
|66. What sort of target user’s performance is feedback to them?|
|67. Is debriefing included in the process of the game play?|
|68. Is the game able to measure level of engagement?|
|69. Is there reward system designed into the game (e.g. post-game reward, dynamic in-game scores, affordance-oriented rewards, etc)?|
|70. Was a clear motivational strategy adopted by the game (e.g. goal setting, problem solving, plot change, etc)?|
Stage 5: Game Accreditation
The validation framework listed above is a set of guidelines that guides the serious game team members (domain experts, instructional designers and game developers) in designing and developing their games. However, the serious games industry would still need a set of independent accreditation criteria to rate the games for its readiness for the market. This is especially helpful when organisations depend on a set of unbiased rating for the games to be deployed in their training curriculum.
Thus, we adapted the Dutch Society for Simulation Health (DSSH) Validation Framework by Dankbaar et el as the Accreditation Model in SG-DiVA Framework. The DSSH Serious Games Validation Framework described 62 items in 5 main themes (Table 1), aimed at assessing a serious game’s rationale, functionality, validity, and data safety. It does not aim to assess its effectiveness in terms of success or user attractiveness which should have been done at the validation stage. Thus, this set of DSSH Serious Games Validation Framework is deem more appropriate to be a set of accreditations that provides an indication of the serious game market-readiness.
Stage 6: Implementation
On the basis of the findings of stage 4, the serious game may wish to further refine and re-evaluate updated versions of the tool or proceed immediately with rollout toward the intended target audience. Regardless of the outcomes of the game evaluation studies, developers should try to disseminate the study findings to the wider serious games research community, as this will help further advance the field. Even publishing and communicating null results may provide insight on how to optimize serious games interventions and provide guidance on the best practices and pitfalls to avoid. If a game is successfully validated and implemented or marketed for the intended target audience, efforts should be made to collect user data in the field, to help monitor for adverse events (if relevant), or to further explore the validity and use of the serious games.
To ensure market readiness and effectively implement and roll out the serious games to the market, a business expert should be consulted, ideally no later than stage 3. Aside from the business expert, it may be relevant to consult with regulators, industry professionals, industry-specific assessment agencies, and others throughout the various stages of the development process. Working with multiple stakeholders carries a specific set of challenges such as clear alignment on roles and responsibilities, finding a common language, understanding of limitations, and consensus on priorities and outcomes.