Have you already registered for our 4th FLIP+ annual event taking place next week on 10th-11thJune, from 12:00 – 14:00 CET?
Join us to learn about the opportunities and challenges for IEA studies (TIMSS, PIRLS, ICILS, ICCS), as institutions experience a transition from paper-based to computer-based assessments (CBA): how are they dealing with item authoring, translation, translation verification, national adaptation and test delivery? What about institutions that choose to administer both paper-based and CBA? How do these assessments change the reporting of the results of the assessments? IEA will share the perspective of how the international study centres coordinate international large-scale assessments.
The DIPF | Leibniz Institute for Research and Information in Education will then briefly present selected infrastructure work and research activities related to computer-based assessments. Reporting from a high-stakes assessment based on TAO, DIPF will share with participants, the development of the CBA Item Builder, an authoring tool for complex items that can be used to implement or extend assessments. Using a mixed-mode assessment as an illustration, you will learn how personal interviews may be replaced by online web-based assessments. In addition, DIPF will share tools for the analyses of log file data, tools and microservices for assessment-related data processing such as NLP tasks and test assembly problems. DIPF will outline ongoing and future work for changing from assessments of learning to assessments for learning.
Other institutions will also briefly present their e-assessment work, including adaptations related to changes due to the pandemic. These include the National Institute for Evaluation (INVALSI) of Italy, the Centre for Educational Assessment (CAEd) attached to the Federal University of Juiz de Fora, Brazil, the National Agency for Education (Skolverket) in Sweden, the National Assessment & Exam Centre (NAEC) of Georgia, the CITO Foundation in the Netherlands and the DEPP from the Ministry of Education in France.
On both days, the FLIP+ team will also provide participants with an update on the ongoing work related to the Item Library, in particular a first design of the minimum viable product created by the CITO team, based on the technical specifications provided by the FLIP+ Item Library team.
Over a month ago, we presented to you observational insights from recent user experience (UX) studies. Our next webinar coming up this Friday, 23rd April at 16:00 CET, will be addressing this same topic but within the context of findings from a Ph. D. thesis. A team from the University of Luxembourg will present factors influencing pupils’ user experience with a Technology-Based Assessment (TBA) application.
By applying a mixed-methods approach (i.e. user observation, self-developed questionnaire), the team looked into the interplay of different factors on a contextual and personal level, affecting the UX with a TBA application. The findings are based on two case studies in France and Luxembourg and present shortcomings of the current TBA design. We also shed light on measuring childrens’ UX by, for example, discussing weaknesses of the Smileyometer, a prominent scale design for conducting UX research with children.
This research was carried out within the Human-Computer Interaction Research group of the University of Luxembourg and in collaboration with the Luxembourg Centre for Educational Testing (LUCET) and the Department of Student Assessment (DEPP) within French Ministry of Education.
Our fourth annual event will be held on 10th-11th June 2021, from 12:00 – 14:00 CET on both days.
The theme would centre on countries’ use of technology in assessment in response to the COVID-19 pandemic. Sub-topics to include are high-school assessment, learning losses due to school closures and lockdown; sharing of experiences (templates to monitor changes in what is done in schools / education), digital tools used and content; how can FLIP+ Item Library be used to help in these circumstances.
The agenda on the first day will include country experiences on main e-assessment developments and adaptation due to the pandemic. An update on ongoing work will be shared on the second day, related to the Item Library Content (Maths and English) with, if necessary, parallel breakout sessions according to themes to be defined ahead. A further update will be provided on the platform development (update, standards, technology…). Further details to come soon.
To sign up for the event, please REGISTER HERE
For all members of FLIP+, please note that as our General Assembly cannot again be held in person this year, the association will organize it virtually on Thu 15th April 2021 from 13:00-14:00 CET. An invitation and agenda will be sent to all members by the beginning of next week.
Mar 12th – User experience in computer-based assessments, by Dr. Bryan Maddox, Assessment MicroAnalytics™
User Experience (UX) studies are an essential part of good test development. In computer-based assessments, they are all the more important because of the big investments that are necessary in item and platform design, and because of the potential for dimensions of the digital mode of assessment to unexpectedly impact on test performance and validity.
In this FLIP+ webinar, Bryan Maddox will share observational insights from recent UX studies in computer-based assessment. He will describe how UX data is collected, the diverse ways that test takers engage with, and navigate compute-based test items, their cognitive and affective strategies, and their responses to success and failure. This will include evidence from recent UX studies involving data from eye tracking, gesture, facial recognition, and think aloud protocols.
Bryan will highlight some of the strengths and hazards of computer-based assessment design and share tips on how UX studies can strengthen student engagement, support construct validation, and enhance the use of process data.
Mar 19th – The taxonomy and framework for digital assessment items – International Baccalaureate and CITO
In this webinar, Rebecca Hamer, from the International Baccalaureate (IB) and Caroline Jongkamp, from CITO, will present their work related to a framework for categorising digital assessments inspired by Kathleen Scalise’s 2009 taxonomy. The framework builds on several workshops and includes many innovative digital item types.
Participants will be introduced to the framework and some (preliminary) results will be presented. The second phase to validate the framework will be described, with an invitation to all for further contributions to this work. In addition, first thoughts will be shared on whether the framework would help systematise the meta-tagging in the FLIP+ Item Library.
Jan 29th – Innovations in Digital Assessment, by CITO, Netherlands
In this webinar, the CITO team will show you examples of innovative assessments that they developed for primary and secondary education in the Netherlands.
The constructs of the assessments vary widely (speaking, reading, digital literacy, empathetic ability) but what they have in common is the way they incorporate playfulness and direct measurement into the assessment.
Above all, the discussion will focus on the consequences of these developments in the context of assessment validity and how it affects the (technical functionalities of) the testing environment.
Feb 5th – A new diagnostic reading literacy test and online scoring and reporting tool, by Educational Research Centre (ERC), Ireland
This webinar will be presented by the Educational Research Centre (ERC) and is a collaboration between the ERC and the National Educational Psychological Service (NEPS) in Ireland.
The aim is to share their work in providing a standardised diagnostic test of literacy to schools. This paper-based test, the PPAD-E (Post-Primary Assessment and Diagnosis – English) was developed and piloted by NEPS and standardised by the ERC using a nationally representative sample of Grade 7 students in Oct-Nov 2019. The complete test package includes five subtests: word reading, spelling, reading speed, reading comprehension and writing samples.
The webinar will provide details on why and how the test was developed and standardised. It will also cover the advantages and benefits of the test, its structure and content as well as the training and support materials for teachers. Participants will also be able to view a demo of the scoring and reporting tool (features, functionality, security).
Feb 26th – Assessment of social-emotional skills, by Instituto Ayrton Senna, Brazil
In this webinar, participants will have the privilege of learning about a framework to understand and measure social-emotional skills, developed by the Institute of Ayrton Senna in Brazil. Social-emotional skills are individual characteristics that are manifested in consistent patterns of thoughts, feelings and behaviours. They influence socioeconomic outcomes throughout someone’s life and they are considered pivotal to life and work adaption, for example. This webinar will cover how the institute’s projects tackle the idiosyncrasies of each targeted population.
Can’t attend live? We still invite you to sign up, as a recording of the webinars will be made available to all those who have registered.
We have had a great turnout for our webinars that were organised in December 2020 and last week. A big shout out to all participants who joined and actively engaged in the discussions.
The upcoming webinars include an update of the development work undertaken by the FLIP+ team with regards to the FLIP+ Maths Item Library, a showcase of innovations in digital assessment by CITO, Netherlands and a new diagnostic reading literacy test and online scoring and reporting tool, to be presented by the Educational Research Centre in Dublin, Ireland.
On December 11th, participants were able to learn about the INVALSI formative testing project, which provides schools with tools for diagnostic (initial) and ongoing assessment of student achievement on the expected proficiency and content goals for the previous school grade. The assessment also aims to provide concrete and operational help to schools, teachers, students and their families to reduce the disadvantages that have been exacerbated as a result of remote teaching. This initiative comes in the light of the pandemic affecting schools, and INVALSI intends to make use of the vast accumulated knowledge gained over more than ten years of census-based standardised tests, in grades 2, 5, 8, 10 and 13.
On December 18th, Denmark presented a joint assessment tool based on the principles of design thinking, which is an approach to tackling problems and thereby gaining so-called 21st century skills. During the webinar, interactive items were shown live and participants were able to learn about examples of tasks and how human and automatic coding are performed. The joint assessment tool presented by the Danish School of Education, Aarhus University, aims to find out to what extent the interventions impact students’ proficiency in design thinking.
Last Friday (January 8th), the FLIP Development team presented its work on e-assessment tools that are being developed in different member institutions. Again, interactive items and tools for 21st century skills, maths and science as well as literacy and numeracy were presented.
Participants to the webinars have expressed great satisfaction with all the content so far and the FLIP+ team is very grateful for all the collaboration and sharing that is taking place within the association.
During our 3rd FLIP+ annual event 2020, many participants expressed interest in learning about the ongoing developments in e-assessment within our FLIP+ e-assessment community. In response, our first three webinars on Zoom will cover the experiences of Italy, Denmark as well as the items and tools being developed by our FLIP+ Development team. The webinar dates are as follows:
The webinars will last an hour. At the end of the presentations, there will be some time for questions and sharing.
Can’t attend live? You should still register as we will be sending out the recording after the webinar to all registrants.
During its third international FLIP+ online event last week, the FLIP+ association gathered over 100 participants from 50 different institutions working in the field of educational assessment. Participants joined in from 24 countries as well as from IEA and OECD.
Under the FLIP+ motto of “sharing”, participants were able to learn about the e-assessment experiences of countries like Denmark, Georgia, Lithuania and Norway and gain insights about developments in six other FLIP+ member institutions. Work undertaken this year related to the FLIP+ Item Library was also presented. It should be recalled that the first ideas to set up the item library were initially discussed during the 2019 annual event in Rome. A year later, participants greatly welcomed the live demo of the initial prototype of the Item Library platform which showed where the items would be stored, how they could be accessed, reviewed, worked upon and shared.
Great feedback was received through the active online discussions on both days and through a mini satisfaction survey about the event. Links to test portals, national item resources and articles were shared, and institutions reflected on how to further contribute to the future work of FLIP+. The next milestone will be the definition of FLIP+’s formal collaboration model for item sharing. Following the suggestions of the event participants, FLIP+ will also be organising a series of webinars from September 2020 onwards. This will include topics such as assessment of design thinking (21stC skills), digital tools related to complex maths items, speech recognition and marking, taxonomy of digital assessment items, user experience studies in e-assessment, digitalising exams, translation workflows, and in-house development of open-source proctoring solutions. Tune in to this space for more details.
The presentations and shared resources from the 2-day FLIP+ event are available here:
On the first day of our online event, attendees will be able to hear the e-assessment experiences of FLIP+ member countries. These include the assessment of design thinking in Denmark, piloting e-assessment of items and an online delivery system in Georgia, the Lithuanian experience of transiting to e-assessment of literacy and how Norway is digitalising its national exams. Read on for more details. Other member countries and institutions will also update us on the ongoing e-assessment activities this year. To join our event, please register by completing the registration form here.
Denmark: Assessment of Design Thinking
Two ongoing Danish research projects have constructed teaching units building on the principles of design thinking, which they propose as an approach to tackling problems and thereby gaining so-called 21st century skills. The joint assessment tool will serve to estimate an effect of the interventions on the students’ proficiency in design thinking.
The test measures four dimensions of Design Thinking: EMPATHY (The ability to understand the needs and perceptions of others – end users as well as peer collaborators), IDEATION (The ability to draw on past experience to come up with many, different ideas), MODELING (The ability to understand, construct and critique representations of ideas and information) and PROCESS MANAGEMENT (The ability to prioritize as well as determine temporal order, duration and simultaneity of tasks and activities in order to inform design decisions).
Five new PCI’s are developed: Brainstorm, Voxelcraft, Route, Room, and Gantt. The presentation will give examples of tasks and how human and automatic coding are performed.
Georgia: Assessment for Development
How do we build an assessment that guides educational decision-making on different levels? How can we use technologies to leverage the use of assessment data for the improvement of teaching and learning?
Assessment for Development is an e-assessment project in Georgia which aims to design and generate assessment data that guides the improvement of teaching and learning on different educational levels.
The assessment is fully online. Items are created and delivered to the students via the TAO testing platform. The results are collected and analysed through in-house built e-reporting system. E-reports provide elaborated analysis on the strength and challenges for each student, as well as teaching and learning trends on classroom and school levels.
The project was piloted in February 2020 with 6 000 students so as to trying out the items and online test-delivery system. The next pilot is planned to test the e-reporting module and its compliance with e-delivery and e-scoring modules.
Lithuania: Transition to e-Assessment: a tool to strengthen e-Literacy
Founded by the Ministry of Education, Science and Sport of the Republic of Lithuania, The National Agency for Education (NSA) is tasked with implementing education policies, ensuring the quality of education and monitoring, researching and implementing educational solutions.
The transition towards e-assessment is related to the national context of education and highlights the tradition of paper-based assessment, expectations and reactions of community. The experiences over the past two years and that of the recent lockdown in Lithuania confirmed that the chosen priority for e-assessment technologies was the right solution. However, it is still necessary to keep in mind that there are different approaches of stakeholders, the need to ensure balance between the novelty of technology and communication, strict planning and attention to details. Lithuania struggles not for the bigger numbers of e-assessment participants, but rather to create and implement the national digital strategy with professional and adaptive e-assessment system.
Norway: Digitalising exams in Norway
In this presentation we will show how Norway have digitalised exams since 2007 using a system based on the distribution of PDFs. From 2020 we will start truly digitalising exams using the TAO testing platform. Some of our perspectives in this transformation are outlined here.
How do we engage students and teachers as well as exam authors in developing new digital exams, to ensure the human perspective in digital transformation? We will point out some challenges of digitalisation of exams in different subjects. How do we construct exams that utilize the possibilities in our new digital platform and still maintain the traditions of the subjects and reflect the goals of the new curriculum?
To us, maintaining the qualities of our current exams and ensuring a quality increase on relevant areas such as psychometrics and analytic assessment is crucial and must be reflected in the design of the new framework for exam creation that we are developing.
Is your institution currently transiting from paper-based to e-assessments, or are you well on the way in your journey to create large-scale e-assessments for students? Would you like to hear how other institutions in this field are building digital assessment tools? Would you like to learn how a group of e-assessment experts are currently creating a digital item library of mathematics items?
The FLIP+ e-assessment association is pleased to invite you to its 3rd international event on 11th – 12th June 2020. Due to the actual global circumstances, the event will be held online via Zoom from 11:45 to 14:30 CET on both days.
The presentations will include country experiences from Denmark, Georgia, Lithuania and Norway, FLIP+ member updates on ongoing developments related to e-assessment progress, the ongoing work carried out by members on both the FLIP+ Item Library and the technological developments related to the platform. The agenda is found here.
If you wish to participate in our online event, please register by completing the registration form. You will receive the access detail upon registration. For further information please do not hesitate to contact us at firstname.lastname@example.org.
For all members of FLIP+, please note that as our General Assembly cannot be held in person this year, the association has opted to organize it separately during a remote meeting on Thu 25th June 2020 from 13:00-14:00 CET. An invitation and agenda will be sent to all members by the beginning of June.
Due to the global impact of the unfolding coronavirus (COVID-19) crisis, the FLIP+ association has decided to switch its third in-person international event to a remote meeting on 11th – 12th June 2020.
The virtual space will be used to present country experiences, progress on the work carried out by members on both the FLIP+ Item Library and the technological developments related to the platform.
A link for registering to the event, as well as a more detailed agenda, will be available soon.
During the 2nd thematic FLIP+ workshop on the 7th February, 2020, held at INVALSI, Rome, participants from eight countries addressed the issues related to the construction of the item library. The goals were to identify members who would work on the public content and the technical implementation of the item library.
Following the presentation of some concrete examples from the Italian context, there was an in-depth discussion between participants related to:
Participants then teamed up in three subgroups and three parallel sessions were organised to discuss: the item library catalogue, the technical creation and implementation of the platform underlying the item library as well as the content itself. A collective summary record of each plenary session will be produced and shared with all participants.
Following our kick-off item-library workshop at the end of October in Rome, a first 1-day working meeting will be organized on Friday 7th February, 2020 in Rome to begin concrete work on creating the content of the item library (IL).
This will include outlining the IL structure and defining both the conceptual framework underlying the IL as well as the structure of the IL catalogue. To start off the process, participants in this working group will focus on mathematics as this was considered as a transversal need for all FLIP+ members.
The intention is to present the first product of this working meeting during the 3rd international FLIP+ event to be held on 11th-12th June (venue to be announced soon).
Should you wish to participate in this working group on item library content or for more information, please write to us at email@example.com by 10th January 2020.
The FLIP+ Steering Committee members met for a one-day meeting in EduPôle, Walferdange-Luxembourg on the 12th of December 2019 to discuss the organization of the planned work for creation of the item library, as well as the agenda of the 3rd international FLIP+ event in June 2020.
During this meeting, the FLIP+ Steering Committee elaborated on three different work packages which would cover targeted actions to be undertaken by three working groups, in order to build the item library. These refer to the item library content, the general organization of the item library and the technology underlying the item library. FLIP+ members will receive an invitation in the coming weeks to participate in these different working groups. Webinars will also be organized to current developments related to e-assessment in member countries.
The FLIP+ e-association welcomed around 40 participants on Monday 28th October in Rome to its first workshop, with the aim of discussing the creation of its own digital item library which would be shared eventually by all association members. The participants expressed the need for members to share and pool all the knowledge and experience related to concepts, frameworks, format, technology, content, skills existing in their institutional context. This would serve as a basis to identify new interactive items to be built and at a later stage piloting new experiences being tried out in member countries. Discussions included the general principles related to the item library such as the collaborative development of items, the conditions of access and use of the items as well as the use of the items by different stakeholders. The various aspects to be considered for the creation and maintenance of the item library were also addressed, such as the concepts, approaches, frameworks underlying item bank systems and the technology requirements as well as item standards and formats. First ideas were put forward on how to organize the work related to the creation of the item library into different work packages. Participants also had the opportunity to learn about and discover an example of the Italian item repository.
During the 2nd international FLIP+ event in Rome in June 2019, participants expressed a strong need to create a shared item library for the FLIP+ e-assessment community. To this effect, a one-day kick-off meeting will be organised on Monday 28th October in Rome. The objectives of this first workshop is to address the philosophy and functioning of the item library, the requirements of the underlying technology (definition of security, access, user roles, …), the rules and modalities of sharing, the usage of the items and the implications of this sharing. Following the results of the workshop, a follow-up working meeting will be organised to start the actual creation / sharing of the items.
In order to prepare for the workshop, a support document will be sent out to participants in October, outlining the position of FLIP+ with regards to the philosophy of the item library as described above. Participants are expected to contribute actively by reflecting ahead on aspects such as the contents their institution can potentially add to the library, the contents they expect to benefit from, the expertise or technical know-how they (or their institution) can offer to create the library and its content. The creation of such an item library is an ambitious endeavour which can only succeed through the association’s motto of “sharing”.
As we announced before the summer holidays, the FLIP+ community is now ready to welcome new members. The annual membership fees are €1000 for an institutional member and €300 for an individual member. To join the association, simply write to the FLIP+ e-assessment secretariat at firstname.lastname@example.org, providing the following details:
The FLIP+ secretariat will send you an invoice and the fees can be transferred to the bank account of the FLIP+ e-assessment association. Any queries regarding the membership enrolment should be addressed by e-mail email@example.com to the association’s treasurer, Thierry ROCHER. Please note that the annual membership fee should be settled by the 31st December of the current year.
The FLIP+ e-assessment community is proud to announce that it is now an officially-established non-profit association.
This is a great milestone for our initiative which grew from informal thoughts and first reflections in 2017, followed by small but bold concrete steps in the years after, and to the launch of our e-assessment association today. It is a formal start to a great adventure with all those who seek to share and collectively develop innovative digital solutions to traditional assessment design, instruments and test administration to improve assessment of and for learning.
Applications for membership will be open as at September 1st 2019.
The Association has three categories of members:
Stay tuned for more details on the statutes of the association and on how to join as a member.
The second international FLIP+ event was held in Rome on 6th – 7th June 2019. Over 50 participants from educational assessment and technology institutions in 13 countries were present. Representatives from the IEA, OECD and UNESCO also attended the meeting.
The event offered the opportunity for participants to share their experiences related to e-assessment. The FLIP+ team presented updates on e-assessment activities in their countries as well as current and planned technological developments (such as item developments, offline solutions, e-assessment portal, functionalities, item banking, tools, data analytics).
Other than sharing knowledge and experiences, the FLIP+ team facilitated a session to explain the value-added of being part of the FLIP+ e-assessment community and the broad principles underlying the collaborative framework of FLIP+. Participants expressed a strong need for a shared item library for the FLIP+ e-assessment community. In line with this reflection, a separate FLIP+ workshop will be organized in Rome in autumn 2019 to address the modalities and principles underlying the FLIP+ item library. Following the brainstorming session in the first event in Paris in 2018 regarding the governance structure of FLIP+, a session was dedicated this year to presenting the statutes of the future FLIP+ association. In parallel, the FLIP+ developer community held separate conversations to share updates, demos and functionalities.
The presentations from the 2nd international FLIP+ event are available for download:
All participants actively participated in all the sessions and the FLIP+ team strongly believes that this sharing within our community can only contribute to strengthening the world of e-assessment in education. The FLIP+ team will organize its third international event on the 11th-12th June 2020. Watch this space for more detail and for news related to the upcoming FLIP+ activities.
The FLIP coordinating team met for a one-day meeting on the 25th of March 2019 to draw up the legal aspects of the collaboration. In order to identify the initiative with the idea of openness and common shared interest in the field of electronic assessment, the partnership was renamed the FLIP+ e-assessment community.
During this meeting, the team also discussed the road map priorities related to technical developments and outlined strategies for sharing content. The agenda of the upcoming FLIP+ event in Rome in June 2019 was discussed in detail. Invitations to the event, a draft agenda and the instructions for registration will be sent out by mid-April.
The second FLIP event will be held in Rome on the 6th – 7th of June 2019. More information and registration form will be available soon!
A FLIP-developer community was set up in January 2019 in order to exchange information and share experience on technical aspects of the e-assessment, as well as to share progress on joint FLIP developments. Up to now, two joint developments – an offline e-assessment solution and an app solution (both based on TAO) – have been undertaken by three member countries: France, Portugal and Brazil.
The FLIP coordinating team met in Lisbon on 5th-7th of December 2018 to further discuss the legal and technical aspects of the partnership and to plan the next FLIP event that will take place in Rome in June 2019.
The 1st FLIP event took place in Paris on 18th – 19th of June 2018. Altogether 55 participants from 17 different countries took part in this meeting. This wide interest indicated a promising start to a fruitful and long-lasting community.
This first FLIP event provided a great opportunity for rich inputs and exchanges between participants. Country reports from current FLIP members France, Luxembourg, Italy, Portugal and Brazil were presented.
This was followed by an introduction to an open standard ecosystem IMS and a demonstration of an open source assessment product TAO.
Multiple extended roadmap sessions were then held, addressing issues such as assessment platform functionalities, security and data protection, data analytics and reporting, item banking and technology enhanced items, etc.
A first brainstorming regarding the legal organization of the partnership and the setting-up of a technical community was done during the two community-building sessions. The intention is to draw up a more formal framework that would define FLIP’s governance structure and mode of functioning as well as detailing how costs would be shared.
Feedback from the participants of the event showed an overall satisfaction and interest towards the event and the initiative. All participants responding to the feedback survey found the content of the first FLIP event programme interesting and varied and the organization of the first FLIP event entirely satisfactory. Participants also found the first FLIP event to be professionally useful.
75% of the respondents noted that they were already involved in the FLIP partnership or willing to play an active role in it in the future.
The participants showed most interest towards the following developments: portal for reporting assessment results, (online) and interoperable marking system, and technology-enhanced items.