Andreia Inamorato dos Santos
The Open University - Institute of Educational Technology
a.i.santos@open.ac.uk
A- Formação de Profissionais para Educação
a Distância
2- Educação Universitária
Resumo
Esta apresentação visa a discutir práticas
de avaliação comuns a ambientes de aprendizagem virtuais (VLEs)
para o ensino de adultos em cursos de pós-graduação. Avaliação
pode ser uma ferramenta poderosa para forças contraditórias –
tanto para facilitar ensino-aprendizagem como para fazer o papel de “força
disciplinar” que irá estabelecer as regras sobre o que pode e deve
ser feito numa sala de aula virtual. A maioria dos ambientes de aprendizagem
virtuais tem mecanismos para localizar o número de acessos e mensagens
postadas pelos alunos. Este tipo de mecanismo, quando usado com o propósito
de avaliação , pode ser um causador da dinâmica de relações
de poder entre o tutor e os alunos (Land & Bayne, 2001). Outras estratégias
de avaliação, como a avaliação em grupo de ensino
colaborativo, também podem gerar um clima de vigilância; uma abordagem
panoptical para o ensino, que busca a participação e compromisso
dos alunos - usando os termos de Foucault (1969). Avaliações e
tarefas de aprendizagem pré-desenvolvidas e prazos curtos para a submissão
de trabalhos também podem ser a causa de tensão e disconforto
nesses ambientes.
Esta pesquisa baseia-se numa revisão de literatura em técnicas
de avaliação online e também num estudo de caso realizado
na Open University, no Reino Unido. Esta apresentação vai sugerir
práticas alternativas para a avaliação de alunos em VLEs
visando a uma abordagem mais democrática do ensino e enfocando o uso
das novas tecnologias com o propósito de avaliação.
Palavras-chave:
avaliação online, ambientes de aprendizagem virtuais, VLEs,
aprendizagem colaborativa, poder disciplinar.
Abstract
This presentation aims to discuss common assessment practices in
virtual learning environments (VLEs) for the teaching of adults in academic
settings. It highlights some contradictions between teaching practices and assessment
procedures.
Assessment can be a powerful tool for contradictory forces - both for
enhancing teaching and learning and for playing the role of a “disciplinary
power” that will set the rules of what can and should be done in a virtual
classroom. Most VLEs have built in mechanisms to track the number of accesses
and posted messages made by students. This facility, when used for assessment
purposes, can be a trigger for the dynamics of power relations between the tutor
and the students (Land & Bayne, 2001). Other assessment strategies, such
as group evaluation of collaborative work can also set up a climate of surveillance;
a panoptical approach to teaching that seeks for students' participation and
commitment, in Foucauldian terms (Foucault, 1969). Pre-designed assessments
and learning tasks, and tight deadlines for work submission can also be the
cause of tension and discomfort in such environments.
This research is based on a literature review of online assessment
techniques and also on a case-study set up at the Open University, UK. This
presentation will suggest alternative practices for assessing students in VLEs,
envisaging a more democratic view of learning and focusing on the use of the
new technologies for assessment purposes.
Key words
online assessment, virtual learning environments, collaborative learning,
disciplinary power.
Introduction
Virtual Learning Environments, as face to face courses, offer a number of possibilities for assessment _ electronic interactive assessments (EIA), collaborative group tasks, students' participation in synchronous and asynchronous communication, and the submission of electronic documents via the famous drop in box of virtual learning environments. Online tutors have many choices on how to assess the students, considering that the new technologies offer the possibility of carrying out both summative and formative types of assessment within the same environment.
The issue regarding assessment, though, is usually not seen as a teaching matter. The question 'Do I use assessment as a teaching method or a teaching method for assessment?' is not commonly addressed appropriately during the design of courses. Assessment as a teaching method acknowledges ontologies but sees them as a part of a bigger and unique set of knowledge about the world, which works in a spiral, comprehensive and intertextual way. It means that assessment is then an important opportunity to check what has been learned, what needs to be taught or addressed, the ways in which the course goals can be achieved and what it takes to achieve this goal. In this sense, the assessment is an opportunity to recognise that any knowledge acquired is in inter-relationship with many other bodies of knowledge, and that the assessment does not mean the end of the learning journey. Assessment in this perspective is seen as a part of a teaching procedure, a time for reflection and renegotiation of learning procedures and goals.
A teaching method for assessment, on the other hand, is that familiar view of evaluation in which the students have to "refund" the institution with all the knowledge that has been "paid in" to their academic accounts - the famous banker's conceptualisation of education described by Paulo Freire (1988). Teaching methods, in this sense, mean every teaching practice that aims to achieve academic goals and focuses on the ‘final exam’. Students can also assume this type of attitude as for their learning approach, when they study ‘focusing the exam’ and not on ‘what they have been learning’ and ‘how they can apply" this learning. It is not to ignore the fact that evaluation plays a major role in driving student learning appropriately (Knight, 1995), but it is important that assessment is not seen as the end of a learning process and its ultimate goal, but instead as a process which needs to be treated as a conduit that will lead to further personal fulfilment and achievement of goals that have been previously established by the student for their professional life.
Online courses, in the context of assessment, bring about the necessity to rethink existing practices. Weller (2002), points out that the change in pedagogical practices brought about by the new technologies requires new methods of assessment. He suggests that formal exams were designed to suit the need of traditional face to face academic contexts; and since distance education courses are different in the way they are designed, the use of traditional exams should be revisited. Among the assessment methods that could be adopted as alternatives for online courses, the compilation of portfolios, peer assessment and the development of student websites were signposted by Weller as valid options. Macdonald (2003) suggests that the current agenda for higher education is one based on ‘skills’, and that therefore not only must the assessment be appropriate for the subject content of the course but it must also play an important role in supporting course pedagogy. Assessment, then, should not be a contradictory practice to the methodological approach chosen for teaching and learning process, but it should be a continuum of the application of such approach, if coherent discourses and practices are to be pursued.
The collaborative learning and current assessment practices
The growing use of 'collaborative learning' as the core approach for teaching and learning online can assume contradictory versions when assessment is concerned. Collaborative learning is an approach that has grown out of constructivist principles and its premise is that learning is a social process (Weller, 2002). According to Weller, constructivism is a learning theory rather than a teaching approach, so it can be realised in terms of teaching in a variety of ways, and the collaborative approach is one of them. Constructivism, like objectivism, holds to the principle that there is a real world that can be experienced, however the argument is that meaning is imposed in the world by us, rather than existing in the world independently of us (Duffy and Jonassen, 1992). Therefore constructivism emphasises the social construction of knowledge, and a collaborative approach to learning will focus on group work and the joint construction and validation of such knowledge.
Weller (2002) claims that the reason why constructivism is so popular as an approach to teaching online is that it suits the advantages the Net can bring to distance situations, by placing less emphasis on the educator and more on the learner. His personal feeling about collaborative learning, though, is that "it should be implemented with caution, because the requirement for students to work collaboratively runs counter to the flexibility offered by distance study, since students are tied to specific timing of activities". An example of this can be drawn out of the following postgraduate student messages posted in a discussion board in 2003 (please note that all students and tutor's names are fictitious in this research):
Hello everybody,
This is Rachel again. I am starting to get desperate to hear from you. I seem to have failed accessing my university's email account and because of this I just don't know what is going on in the group. Have you decided who does what? Can somebody update me, please, I want to contribute.
The reasons for the student to feel so out of context is that the student was based abroad from where the course was running and despite having time zones differences, she could also not join the rest of the group who lived locally and enjoyed getting together to discuss the assignments’ tasks. However, if a course is to aim a global market, its syllabus, tasks, teaching and assessment approaches should reflect a concern for such an issue. It is not feasible to expect students to work well as a team and perform brilliant group assignments if the timing allocated for the building up of group work skills and socialisation is not enough; and if some students can be favoured by the chance to meet personally in order to grow such socialisation.
According to Weller (2002), although the collaborative approach promotes the development of communication skills and the exposure to different ideas, the time taken to perform group tasks can be excessive if there is a lot of debate and negotiation about which roles people will adopt. Weller also claims that the loss of independence could be another issue, as many students prefer to work at their own pace and independently of others. When assessment is concerned, the loss of independency can be problematic for the students. Weller (2002) claims that one of the common complaints relating to assessed group work is that it benefits lazy students, as they can be advantaged if one mark is awarded to the overall group outcome. Instead, he suggests that the asynchronous and self-documenting nature of online courses brings the possibility for allocating marks for the process itself, rather than basing it uniquely in outcome. Allocating a proportion of the marks to the quality of individual contribution could then be an alternative. However, it is during course design that the way in which evaluation will work should be decided. Collaborative group work for assessment purposes could have an element of individual contribution pre-defined, and this could still keep a strong collaborative basis, as Weller (2002) suggests. Peer assessment, for example, could also be another additional technique for assessing online, if students grade and comment other fellow students work anonymously, as suggested by Tsai et all (2001: in Weller, 2002). This technique avoids individuals’ likes and dislikes to be constraints when assessment is concerned.
Some tutors award marks according to the quantity of contributions the student made in the general discussion board of the class, and again this is only a valid practice if the quality of the messages are taken into consideration, as it avoids students sending messages empty of real contribution only for the purpose of increasing the number of times messages were sent under their username and therefore increasing their marks for participation. The real challenge is then how to assess the quality of the messages that are going to be taken into consideration for assessment purposes and the criteria should be discussed with the students.
Student tracking tools: the online “panopticon”
The tracking devices that most commercial VLEs have, such as Blackboard and WebCT, can be a surveillance tool instead of prompting an effective way to analyse students’ performance and course effectiveness (Land and Bayne, 2001). These student tracking tools can tell the pages each individual student accessed, the total number of times the student logged in to the system, which messages were opened and the individual students contributions to the discussion board. Land and Bayne (2001) have even used Foucault's’ (1979) panopticon studies of surveillance in prisons as a metaphor to address such tracking tools in VLEs. The authors claim that surveillance for Foucault is an element of the hierarchical observation which is a key instrument of disciplinary power. However, according to Land and Bayne, hierarchical observation is only one of the instruments which disciplinary power exercises itself – the other two are the normalisation of judgement and examination. Therefore, disciplinary power has a collective effect as it engages the subject in the collection of the data relating to them (Land and Bayne, 2001). In an online environment, when students are concerned with how many times they have accessed the discussion board for the purpose of the tracking system records and when they are assessing their peer students’ work, they are using the instruments of the normalisation of judgment (therefore expected behaviour) and the examination, by simply invigilating themselves and the others in a ‘non-obvious’ way. All this surveillance (to use Foucault’s term) is seeking online “fittingness” and is therefore contributing to the creation of the online student’s identity. Land and Bayne (2001) call our attention to the possibility that the online student maybe be objectified in their virtual construction and that the learner may be, as far as the system is concerned, to some extent constituted by the records of their first and last logins, frequency of logins, number of discussion board submissions and so on. Placing these tracking tools as important elements of students’ evaluation in online environments can then be quite a risky pedagogical practice as these mechanisms are often a one sided and limited view of students’ performance.
The case study that follows below illustrates the question of disciplinary power in relation to assessment procedures, but on the perspective of pre-designed assessments and tight deadlines for work submission.
How online learning can evolve around assessment– an overview of a case study
This case study has been carried out at an Open University (UK) course in 2003, as part of a masters’ dissertation in the Institute of Educational Technology (IET). The course investigated was a postgraduate (masters') degree, in the field of social sciences (E841). It was aimed at an international audience, based mostly in Greece and Italy. The students had books and a study guide as learning resources, and also had some online tutorials and the discussion boards available at FirstClass (online conferencing environment). The discussion board was the main mean of communication between students and the tutor, in addition to the telephone if necessary. The course was analysed from February to May 2003, and the students took the course entirely at a distance. The analysis of the conferences in this course provided the opportunity to find out whether the institutional discourses around teaching and learning were also being played out in this newly adopted virtual learning environment. Discourse, in this research, must be understood as the institutionalised use of language. Discourses are concerned with the interplay between language and social practices; that is, how a discourse is both shaped by, and itself contributes to the shaping of, the social practices it constitutes (Santos, 2003). Another aim was to find out what other discourses interplay with the online teaching discourse at the institution. Hence, this study aimed to show the inter-relationship of institutional discourses with the teaching practices with new technologies.
The methods of data collection were triangulated, aiming to have more diverse data, providing more complete information for the data analysis. The methods of data collection were online observations of course delivery, face to face interview with course chair and online interview with tutor and document analysis (institutional documents regarding teaching practices). As for the data analysis approach, this research has drawn on some principles of Critical Discourse Analysis, which regards “language as social practice” (Fairclough, 1995). Critical Discourse Analysis allowed the identification of the inter-relation of discourses (interdiscursivity) in the Teaching Discourse at the Open University (henceforth OU).
Four discourses were identified in an interdiscursive relation to the Teaching Discourse of the OU, they were: the Discourse of Capitalism, the Discourse of Globalisation, The Discourse of Quality and the Discourse of Openness. In very broad terms, the Discourse of Quality was identified to the extent in which new technologies were incorporated as alternative communication tools at the OU; and also within the assessment-driven nature of the course, that is concerned to keep certain quality standards (quality of learning). The focus on achieving international markets reflect the Globalisation Discourse; the need to attract more students to the institution (and to keep existing ones) reflect the Capitalist Discourse and finally, the flexibility the university claims to offer to students (independent learning) reflect the Openness Discourse. However, it is important to point out that these discourses operate in an interdiscursive-relation, that is, they influence, shape and re-shape one another, resulting on particular social practices.
A finding of this study that matters to the assessment theme of this paper is the notion of “assessment as disciplinary power”. Critical Discourse Analysis enabled the discussion of three elements that are constitutive of all discourses: ideology, power and history. The Teaching Discourse at the Open University is constitutive of these elements, drawing them together in what is considered the “order of discourse” (Foucault, 1970/96) in this particular educational setting. Ideology, power and history are constitutive elements of the OU course materials and pedagogy, as part of the teaching discourse, and to a further extent, in the drive towards the marketisation of higher education which is not an exclusivity of the Open University.
In terms of teaching methods, the analysis of the conferences brought into evidence that the current OU pedagogy is assessment-driven. Despite the independent learning philosophy, which should enable the students to guide their own study-rhythm and make them aware of their learning process, the students feel that the TMAs (tutor-marked assignments) and deadlines are always placing the boundaries and putting pressure on their learning.
Claire writes:
"So, I abandoned any hopes of sending Task 2 and started work on etma02. This sense of panic has not abated. I found the samples for text A and B very helpful but I'm still feeling overwhelmed with what I see as the enormity of the task…only a few days to go and inspite of dedicating hours and hours to this assignment I'm still making slow progress…Has everybody actually finished this assignment? Has anyone started reading the course material for the research project?"
This extract is an example of how students perceive the importance of the assignments. The student claims to be putting a lot of effort into the course but instead of enjoying the learning process she is feeling overwhelmed. This shows that ‘assignments ready to consume’ are not always the best alternative for quality-assurance. The opportunity for content and assessment negotiation is a principle of Andragogy (the art and science of helping adults learn - Knowles:1973/79) and it does not seem to be part of the OU teaching method.
Also, there is a need for this student to certify that others are having the same problems as her. This is why the question "Has anybody finished this assignment?" is a type of reassurance that ‘the problem’ is not her incapability to finish the task appropriately in time, but perhaps an over-demand on the task established during course design. Some replies to this message follows:
Sandra writes:
well, the same goes for me... I began to work on task 2 but I never got around to finishing it because I started worrying about etma2. By now everything has been said on texts A and B of task 2 and I've given up the thing altogether. Sorry about that, hope I'm going to get away with it just this time... it's hard enough to complete the assignment by the 16th....
Berenice writes:
And for me.... I began working on task 2 as well but dropped it quite soon.... an started panicking instead!!!! Just hope too to be able to get the assignment done by next Friday!!
Wish you all good luck!
In Sandra’s extract, she agrees with the main message and says she "worries" about the TMA and also feels it is difficult to finish the task in time. Tight deadlines are once more perceived as factor that creates anxiety instead of assurance of quality teaching and learning. On the tutor's reply below, although it is said that it is not necessary to worry too much, there's still an emphasis on the importance of the assessment (40% of the marks).
Don't panic. As you can see from the samples, the analyses for the TMA don't have to go into a lot of detail. Just take it one stage at a time, and make sure you do a good job on the easy parts - the SFL analysis is only 40% of the marks.
As the assignments are pre-established prior to the beginning of the course, there seems not to be possible for the students to renegotiate assessment with the tutor. The tutor, then, although being responsible for the marking of the students’ assignments, usually has his voice ‘shadowed’ by the voices of the course team, which is responsible for writing the course and deciding about learning tasks and assessments. This is a form of pedagogical power, which disciplines the students and regulates the practice of the tutors; in an attempt to ensure that the course will run in a certain order and keep an aimed level of ‘quality’.
However, it is not to say that assignments and assessment are unnecessary or that they can be negotiated in every sense. ‘Assessment’ in the UK is usually a practice that is defined by governmental and institutional policies, such as final examinations and national standards that need to be achieved. Even though, the assignments that lead to the final examinations in a course are usually defined by the course teams, and to a certain extent, they can be more flexible and negotiated; these assignments constitute the path that will lead to a certain learning outcome; that is aimed to be achieved. Therefore, these assessments should be a negotiable process for both students and tutors, respecting individual differences and learning preferences.
In summary, ‘negotiation’ is the key for the new technologies for teaching to optimise learning at the OU, if a more independent type of learning is really to be pursued. Although it is a fact that constant changes in course materials (specially printed versions) would not be financially feasible for the institution (as the interview with the course chair revealed), online learning does offer the opportunity to renegotiate teaching practices. Instead of being seen as just another technology to support learning, computer mediated communication could be used to promote more negotiation of tasks, content and assessment practices.
Conclusion
As online education is an emerging practice in educational settings, it deserves special attention, mainly on issues regarding students' assessment practices. It has been shown that alternative ways of assessment need to be considered for this new type of pedagogical practice.
Firstly, the use of collaborative activities for assessment purposes need to take into consideration that distance education is in principle a type of study which needs to be flexible in 'time' - therefore it is important that students can still have some independency even when working in groups. Secondly, student tracking tools should be avoided as a surveillance mechanism for assessment purposes, as it can also constrain the flexibility of distance education. In this sense, what could be considered regarding students' messages is the 'quality' of the messages that are posted by the students as opposed to the 'quantity'. Thirdly, the pre-designed and inflexible nature of assessment in distance education should also be revisited; especially when computer mediated communication offers the possibility for negotiating such practice.
References
DUFFY, T and JONASSEN, D. (1992) Constructivism and the Technology of Instruction – a conversation, New Jersey: LEA
FAIRCLOUGH, N. (1995) Critical Discourse Analysis: the critical study of language, London: Longman
FOUCAULT, M. (1970/96) A ordem do discurso, Sao Paulo: Loyola
Foucault, M. (1979) Discipline and Punish: the birth of the prison, Harmondsworth: Penguin
FREIRE, P. (1988) Pedagogia do Oprimido, Sao Paulo: Paz e Terra (18th edition)
KNIGHT, P. (1995) Assessment for learning in higher education, London: Kogan Page
KNOWLES, M. (1973) The adult learner: a neglected species, Houston, Texas: Gulf Publishing company
LAND, R. & BAYNE (2001) Screen or Monitor? Surveillance and disciplinary power in online learning environments - http://jimmy.qmuc.ac.uk
MACDONALD, J. (2002) Assessing online collaborative learning: process and product, in: Computers & Education 40 (377-391)
SANTOS, A. (2003) Teaching Methods as Social Practices: a virtual learning environment experience, unpublished dissertation, Milton Keynes: Institute of Educational Technology, The Open University
WELLER, M. (2002) Delivering Learning on the Net - the why, what & how of online education, London: Kogan Page