During its third international FLIP+ online event last week, the FLIP+ association gathered over 100 participants from 50 different institutions working in the field of educational assessment. Participants joined in from 24 countries as well as from IEA and OECD.
Under the FLIP+ motto of “sharing”, participants were able to learn about the e-assessment experiences of countries like Denmark, Georgia, Lithuania and Norway and gain insights about developments in six other FLIP+ member institutions. Work undertaken this year related to the FLIP+ Item Library was also presented. It should be recalled that the first ideas to set up the item library were initially discussed during the 2019 annual event in Rome. A year later, participants greatly welcomed the live demo of the initial prototype of the Item Library platform which showed where the items would be stored, how they could be accessed, reviewed, worked upon and shared.
Great feedback was received through the active online discussions on both days and through a mini satisfaction survey about the event. Links to test portals, national item resources and articles were shared, and institutions reflected on how to further contribute to the future work of FLIP+. The next milestone will be the definition of FLIP+’s formal collaboration model for item sharing. Following the suggestions of the event participants, FLIP+ will also be organising a series of webinars from September 2020 onwards. This will include topics such as assessment of design thinking (21stC skills), digital tools related to complex maths items, speech recognition and marking, taxonomy of digital assessment items, user experience studies in e-assessment, digitalising exams, translation workflows, and in-house development of open-source proctoring solutions. Tune in to this space for more details.
The presentations and shared resources from the 2-day FLIP+ event are available here:
On the first day of our online event, attendees will be able to hear the e-assessment experiences of FLIP+ member countries. These include the assessment of design thinking in Denmark, piloting e-assessment of items and an online delivery system in Georgia, the Lithuanian experience of transiting to e-assessment of literacy and how Norway is digitalising its national exams. Read on for more details. Other member countries and institutions will also update us on the ongoing e-assessment activities this year. To join our event, please register by completing the registration form here.
Denmark: Assessment of Design Thinking
Two ongoing Danish research projects have constructed teaching units building on the principles of design thinking, which they propose as an approach to tackling problems and thereby gaining so-called 21st century skills. The joint assessment tool will serve to estimate an effect of the interventions on the students’ proficiency in design thinking.
The test measures four dimensions of Design Thinking: EMPATHY (The ability to understand the needs and perceptions of others – end users as well as peer collaborators), IDEATION (The ability to draw on past experience to come up with many, different ideas), MODELING (The ability to understand, construct and critique representations of ideas and information) and PROCESS MANAGEMENT (The ability to prioritize as well as determine temporal order, duration and simultaneity of tasks and activities in order to inform design decisions).
Five new PCI’s are developed: Brainstorm, Voxelcraft, Route, Room, and Gantt. The presentation will give examples of tasks and how human and automatic coding are performed.
Georgia: Assessment for Development
How do we build an assessment that guides educational decision-making on different levels? How can we use technologies to leverage the use of assessment data for the improvement of teaching and learning?
Assessment for Development is an e-assessment project in Georgia which aims to design and generate assessment data that guides the improvement of teaching and learning on different educational levels.
The assessment is fully online. Items are created and delivered to the students via the TAO testing platform. The results are collected and analysed through in-house built e-reporting system. E-reports provide elaborated analysis on the strength and challenges for each student, as well as teaching and learning trends on classroom and school levels.
The project was piloted in February 2020 with 6 000 students so as to trying out the items and online test-delivery system. The next pilot is planned to test the e-reporting module and its compliance with e-delivery and e-scoring modules.
Lithuania: Transition to e-Assessment: a tool to strengthen e-Literacy
Founded by the Ministry of Education, Science and Sport of the Republic of Lithuania, The National Agency for Education (NSA) is tasked with implementing education policies, ensuring the quality of education and monitoring, researching and implementing educational solutions.
The transition towards e-assessment is related to the national context of education and highlights the tradition of paper-based assessment, expectations and reactions of community. The experiences over the past two years and that of the recent lockdown in Lithuania confirmed that the chosen priority for e-assessment technologies was the right solution. However, it is still necessary to keep in mind that there are different approaches of stakeholders, the need to ensure balance between the novelty of technology and communication, strict planning and attention to details. Lithuania struggles not for the bigger numbers of e-assessment participants, but rather to create and implement the national digital strategy with professional and adaptive e-assessment system.
Norway: Digitalising exams in Norway
In this presentation we will show how Norway have digitalised exams since 2007 using a system based on the distribution of PDFs. From 2020 we will start truly digitalising exams using the TAO testing platform. Some of our perspectives in this transformation are outlined here.
How do we engage students and teachers as well as exam authors in developing new digital exams, to ensure the human perspective in digital transformation? We will point out some challenges of digitalisation of exams in different subjects. How do we construct exams that utilize the possibilities in our new digital platform and still maintain the traditions of the subjects and reflect the goals of the new curriculum?
To us, maintaining the qualities of our current exams and ensuring a quality increase on relevant areas such as psychometrics and analytic assessment is crucial and must be reflected in the design of the new framework for exam creation that we are developing.
Is your institution currently transiting from paper-based to e-assessments, or are you well on the way in your journey to create large-scale e-assessments for students? Would you like to hear how other institutions in this field are building digital assessment tools? Would you like to learn how a group of e-assessment experts are currently creating a digital item library of mathematics items?
The FLIP+ e-assessment association is pleased to invite you to its 3rd international event on 11th – 12th June 2020. Due to the actual global circumstances, the event will be held online via Zoom from 11:45 to 14:30 CET on both days.
The presentations will include country experiences from Denmark, Georgia, Lithuania and Norway, FLIP+ member updates on ongoing developments related to e-assessment progress, the ongoing work carried out by members on both the FLIP+ Item Library and the technological developments related to the platform. The agenda is found here.
If you wish to participate in our online event, please register by completing the registration form. You will receive the access detail upon registration. For further information please do not hesitate to contact us at firstname.lastname@example.org.
For all members of FLIP+, please note that as our General Assembly cannot be held in person this year, the association has opted to organize it separately during a remote meeting on Thu 25th June 2020 from 13:00-14:00 CET. An invitation and agenda will be sent to all members by the beginning of June.
Due to the global impact of the unfolding coronavirus (COVID-19) crisis, the FLIP+ association has decided to switch its third in-person international event to a remote meeting on 11th – 12th June 2020.
The virtual space will be used to present country experiences, progress on the work carried out by members on both the FLIP+ Item Library and the technological developments related to the platform.
A link for registering to the event, as well as a more detailed agenda, will be available soon.
During the 2nd thematic FLIP+ workshop on the 7th February, 2020, held at INVALSI, Rome, participants from eight countries addressed the issues related to the construction of the item library. The goals were to identify members who would work on the public content and the technical implementation of the item library.
Following the presentation of some concrete examples from the Italian context, there was an in-depth discussion between participants related to:
Participants then teamed up in three subgroups and three parallel sessions were organised to discuss: the item library catalogue, the technical creation and implementation of the platform underlying the item library as well as the content itself. A collective summary record of each plenary session will be produced and shared with all participants.
Following our kick-off item-library workshop at the end of October in Rome, a first 1-day working meeting will be organized on Friday 7th February, 2020 in Rome to begin concrete work on creating the content of the item library (IL).
This will include outlining the IL structure and defining both the conceptual framework underlying the IL as well as the structure of the IL catalogue. To start off the process, participants in this working group will focus on mathematics as this was considered as a transversal need for all FLIP+ members.
The intention is to present the first product of this working meeting during the 3rd international FLIP+ event to be held on 11th-12th June (venue to be announced soon).
Should you wish to participate in this working group on item library content or for more information, please write to us at email@example.com by 10th January 2020.
The FLIP+ Steering Committee members met for a one-day meeting in EduPôle, Walferdange-Luxembourg on the 12th of December 2019 to discuss the organization of the planned work for creation of the item library, as well as the agenda of the 3rd international FLIP+ event in June 2020.
During this meeting, the FLIP+ Steering Committee elaborated on three different work packages which would cover targeted actions to be undertaken by three working groups, in order to build the item library. These refer to the item library content, the general organization of the item library and the technology underlying the item library. FLIP+ members will receive an invitation in the coming weeks to participate in these different working groups. Webinars will also be organized to current developments related to e-assessment in member countries.
The FLIP+ e-association welcomed around 40 participants on Monday 28th October in Rome to its first workshop, with the aim of discussing the creation of its own digital item library which would be shared eventually by all association members. The participants expressed the need for members to share and pool all the knowledge and experience related to concepts, frameworks, format, technology, content, skills existing in their institutional context. This would serve as a basis to identify new interactive items to be built and at a later stage piloting new experiences being tried out in member countries. Discussions included the general principles related to the item library such as the collaborative development of items, the conditions of access and use of the items as well as the use of the items by different stakeholders. The various aspects to be considered for the creation and maintenance of the item library were also addressed, such as the concepts, approaches, frameworks underlying item bank systems and the technology requirements as well as item standards and formats. First ideas were put forward on how to organize the work related to the creation of the item library into different work packages. Participants also had the opportunity to learn about and discover an example of the Italian item repository.
During the 2nd international FLIP+ event in Rome in June 2019, participants expressed a strong need to create a shared item library for the FLIP+ e-assessment community. To this effect, a one-day kick-off meeting will be organised on Monday 28th October in Rome. The objectives of this first workshop is to address the philosophy and functioning of the item library, the requirements of the underlying technology (definition of security, access, user roles, …), the rules and modalities of sharing, the usage of the items and the implications of this sharing. Following the results of the workshop, a follow-up working meeting will be organised to start the actual creation / sharing of the items.
In order to prepare for the workshop, a support document will be sent out to participants in October, outlining the position of FLIP+ with regards to the philosophy of the item library as described above. Participants are expected to contribute actively by reflecting ahead on aspects such as the contents their institution can potentially add to the library, the contents they expect to benefit from, the expertise or technical know-how they (or their institution) can offer to create the library and its content. The creation of such an item library is an ambitious endeavour which can only succeed through the association’s motto of “sharing”.
As we announced before the summer holidays, the FLIP+ community is now ready to welcome new members. The annual membership fees are €1000 for an institutional member and €300 for an individual member. To join the association, simply write to the FLIP+ e-assessment secretariat at firstname.lastname@example.org, providing the following details:
The FLIP+ secretariat will send you an invoice and the fees can be transferred to the bank account of the FLIP+ e-assessment association. Any queries regarding the membership enrolment should be addressed by e-mail email@example.com to the association’s treasurer, Thierry ROCHER. Please note that the annual membership fee should be settled by the 31st December of the current year.
The FLIP+ e-assessment community is proud to announce that it is now an officially-established non-profit association.
This is a great milestone for our initiative which grew from informal thoughts and first reflections in 2017, followed by small but bold concrete steps in the years after, and to the launch of our e-assessment association today. It is a formal start to a great adventure with all those who seek to share and collectively develop innovative digital solutions to traditional assessment design, instruments and test administration to improve assessment of and for learning.
Applications for membership will be open as at September 1st 2019.
The Association has three categories of members:
Stay tuned for more details on the statutes of the association and on how to join as a member.
The second international FLIP+ event was held in Rome on 6th – 7th June 2019. Over 50 participants from educational assessment and technology institutions in 13 countries were present. Representatives from the IEA, OECD and UNESCO also attended the meeting.
The event offered the opportunity for participants to share their experiences related to e-assessment. The FLIP+ team presented updates on e-assessment activities in their countries as well as current and planned technological developments (such as item developments, offline solutions, e-assessment portal, functionalities, item banking, tools, data analytics).
Other than sharing knowledge and experiences, the FLIP+ team facilitated a session to explain the value-added of being part of the FLIP+ e-assessment community and the broad principles underlying the collaborative framework of FLIP+. Participants expressed a strong need for a shared item library for the FLIP+ e-assessment community. In line with this reflection, a separate FLIP+ workshop will be organized in Rome in autumn 2019 to address the modalities and principles underlying the FLIP+ item library. Following the brainstorming session in the first event in Paris in 2018 regarding the governance structure of FLIP+, a session was dedicated this year to presenting the statutes of the future FLIP+ association. In parallel, the FLIP+ developer community held separate conversations to share updates, demos and functionalities.
The presentations from the 2nd international FLIP+ event are available for download:
All participants actively participated in all the sessions and the FLIP+ team strongly believes that this sharing within our community can only contribute to strengthening the world of e-assessment in education. The FLIP+ team will organize its third international event on the 11th-12th June 2020. Watch this space for more detail and for news related to the upcoming FLIP+ activities.
The FLIP coordinating team met for a one-day meeting on the 25th of March 2019 to draw up the legal aspects of the collaboration. In order to identify the initiative with the idea of openness and common shared interest in the field of electronic assessment, the partnership was renamed the FLIP+ e-assessment community.
During this meeting, the team also discussed the road map priorities related to technical developments and outlined strategies for sharing content. The agenda of the upcoming FLIP+ event in Rome in June 2019 was discussed in detail. Invitations to the event, a draft agenda and the instructions for registration will be sent out by mid-April.
The second FLIP event will be held in Rome on the 6th – 7th of June 2019. More information and registration form will be available soon!
A FLIP-developer community was set up in January 2019 in order to exchange information and share experience on technical aspects of the e-assessment, as well as to share progress on joint FLIP developments. Up to now, two joint developments – an offline e-assessment solution and an app solution (both based on TAO) – have been undertaken by three member countries: France, Portugal and Brazil.
The FLIP coordinating team met in Lisbon on 5th-7th of December 2018 to further discuss the legal and technical aspects of the partnership and to plan the next FLIP event that will take place in Rome in June 2019.
The 1st FLIP event took place in Paris on 18th – 19th of June 2018. Altogether 55 participants from 17 different countries took part in this meeting. This wide interest indicated a promising start to a fruitful and long-lasting community.
This first FLIP event provided a great opportunity for rich inputs and exchanges between participants. Country reports from current FLIP members France, Luxembourg, Italy, Portugal and Brazil were presented.
This was followed by an introduction to an open standard ecosystem IMS and a demonstration of an open source assessment product TAO.
Multiple extended roadmap sessions were then held, addressing issues such as assessment platform functionalities, security and data protection, data analytics and reporting, item banking and technology enhanced items, etc.
A first brainstorming regarding the legal organization of the partnership and the setting-up of a technical community was done during the two community-building sessions. The intention is to draw up a more formal framework that would define FLIP’s governance structure and mode of functioning as well as detailing how costs would be shared.
Feedback from the participants of the event showed an overall satisfaction and interest towards the event and the initiative. All participants responding to the feedback survey found the content of the first FLIP event programme interesting and varied and the organization of the first FLIP event entirely satisfactory. Participants also found the first FLIP event to be professionally useful.
75% of the respondents noted that they were already involved in the FLIP partnership or willing to play an active role in it in the future.
The participants showed most interest towards the following developments: portal for reporting assessment results, (online) and interoperable marking system, and technology-enhanced items.