Report 2006:25 R
Follow up of the National Agency for Higher Education´s evaluations of programmes
Overall analysis of this year´s follow-up
The evaluations have stimulated development work
Expected and legitimate evaluation results
The national subject evaluations undertaken by the Swedish National Agency for Higher Education during 2001 and 2001 seem in general to have perceived the problems and strengths of these subjects accurately. Both the general recommendations made by the panels of assessors for each subject and the specific recommendations addressed to the individual higher education institutions are described in the follow-up responses as more or less expected and only in exceptional cases have they been difficult to accept.
The recommendations also seem to have tallied with the adaptations to the Bologna process taking place at undergraduate and postgraduate level in many places.
Internal development work has been stimulated
The majority of institutions emphasise the great benefit that the evaluation has had for their own development work. The self-evaluation process in particular has led to reflection and discussion. Departments that are particularly positive in this respect are those that participated in the evaluations of computer and systems science and also media and communication studies (MKV).
What has been asserted above all about the evaluations involving the major subjects and subjects that are either relatively new or have multidisciplinary features, such as economics, business administration, computer and systems science as well as media and communication studies, is how useful it has been to be able to make comparisons with other departments. It is difficult for teachers and researchers who work with subjects that are spread across many departments to acquire a national overview, while for others who are involved in study programmes that are in the process of being developed and whose contents involve links with other disciplines external inspiration is valuable. “The assessors saw a few risks in the way we worked and in our relationship to long-term funding that we should probably have already been taking more seriously at the time, which shows the value of the outsider´s bird´s eye view in evaluations like these" is how this was summarised by one of the media and communication studies programmes that took part in the evaluation.
Some individual higher education institutions express a desire for more international comparisons. During a period when adaptations are being made to the Bologna process these can valuable.
The evaluations have also been used to improve conditions in individual higher education institutions, as is shown by the follow-up of the evaluations of Latin/Greek/Modern Greek, mathematics, economics as well as Swedish and Nordic languages. The evaluation results have provided new arguments and greater weight in internal discussions about the allocation of resources to the subject, for example, or its organisational parameters.
Several institutions claim, however, it can sometimes be difficult to know if the changes described in the follow-up responses really are the results of evaluation. It is likely that the changes would have been made even without the intervention of the National Agency. The national evaluations have involved a scrutiny that has in its turn prompted, accelerated or provided a basis for measures that would in some cases have been adopted sooner or later in any case.
In a few follow-up responses, a total of around 20, the question is raised of whether the benefits of the evaluations correspond to the cost and the effort that they have involved.
The link between the national image and the images of the specific higher education institutions
On the whole the general analysis made by the panels of assessors of a subject at national level has shown great consistency with their analyses of the conditions that apply for the subject at the individual higher education institutions. The national image and the strengths and weaknesses identified in it have functioned as an overall description of the conditions that prevail at the specific departments.
The follow-up suggests that the higher education institutions have primarily adopted recommendations that were addressed both generally and in the specific analyses.
Small differences between assessment and recommendations
The evaluation reports have generally contained six different types of evaluative judgements: the panel´s assessments of and recommendations for each higher education institution, the panel´s general assessment of and recommendations for the subject, the National Agency decisions and the National Agency´s reflections.[1] Only the National Agency´s decisions have been binding. The higher education institutions have been at liberty to ignore the other assessments.
The follow-up shows, however, that not only the recommendations have been followed. Many of the other assessments of the panels and the National Agency that were not expressed as recommendations seem to have prompted action. The higher education institutions have not always made a distinction between assessment and recommendation.
Entitlement to award degrees rarely questioned
Just over 200 programmes at undergraduate and postgraduate level were evaluated in 2000 and 2001 and the entitlement to award degrees was questioned in the case of only eight of them, four at postgraduate level and four at undergraduate level.[2] The undergraduate programmes for which entitlement to award degrees was questioned were in business administration, economics, linguistics (general linguistics) and Swedish/Nordic languages. The hesitations of the panels of assessors were due to lack of teaching resources at two higher education institutions, lack of research links at a third and low academic standards at a fourth.
The higher education institutions have taken appropriate measures
After their entitlement to award degrees had been questioned, all of the higher education institutions made the improvements considered necessary by the National Agency and the experts. Teaching resources have been augmented at the higher education institutions where they were felt to be inadequate, more teachers have been appointed, more teaching hours have been made available and the research qualifications of the teachers enhanced. One subject states that reorganisation has offered greater scope than before to use teachers from adjacent subjects to teach courses that involve research methodology and supervision for instance. In addition internal reallocation of administrative tasks has made it possible for teachers to undertake more teaching. The experts and the National Agency consider it particularly positive that the action plans submitted by the programmes in Latin/Greek/Modern Greek whose entitlement to award degrees was questioned show that collaboration has been started as one way of compensating for the dearth of resources for these subjects.
In every case the doubts expressed about entitlement to award degrees were withdrawn after one year and no entitlement was revoked. On the other hand a few programmes were terminated after the evaluation. At Göteborg University Modern Greek was no longer offered as an independent subject.
Not all recommendations have resulted in action
Agreement by the higher education institutions with the analyses of the panels of assessors and the National Agency has not always meant that they have taken action based on them. Judging from the follow-up responses, most of the recommendations made by the panels have been followed but not all.
The higher education institutions seem principally to have adopted the relatively simple proposals, i.e. those that it has been comparatively inexpensive to implement and which have not required a great deal of effort from the teachers and other members of the staff. The responses account primarily for measures that it has been possible to adopt within the framework of normal operational development.
The reactions to other types of recommendations have been more varied. Changes that require action by participants who do not form part of the subject setting or the department itself have not always come about. In some evaluations, for instance, the organisational situation of a subject within a specific higher education institution has been questioned and the measures proposed by the panel of assessors have not infrequently affected other departments or faculties and required action by the institution´s central administrators. In these cases it has not been unusual for more weight to be given to internal considerations and regulations than to the National Agency´s evaluations, one reason being that these merely take into account the conditions that apply for individual subjects.
Nor has action always been the outcome of recommendations that require the involvement of participants from outside the higher education institutions themselves, mainly other higher education institutions. In all the evaluations, for instance, the panels of assessors and the National Agency have advocated greater efforts to attain national coordination. These recommendations seem mainly to have resulted in the initiation or enhancement by specific institutions of their collaboration with other individual institutions, not infrequently on the initiative of individuals or groups. More extensive coordination arrangements that encompass several higher education institutions and have been of general significance for the subjects have not come into being as often. There has not been sufficient incentive to stimulate widespread joint endeavours. The exhortations of the National Agency and of the panels of assessors have not carried enough weight.
Another reason why certain recommendations have not been followed is that the higher education institutions have not considered they could afford them. Recommendations that require a great deal of expenditure seem rather to have been perceived as provocative at a time when the institutions found themselves in straitened economic circumstances.
Some higher education institutions have refrained from adopting measures as they have opted to wait for decisions to be made in connection with the ongoing Bologna process, for instance about the new degree ordinance.
Finally, the way in which recommendations have been worded has also played its role. Value judgements that were far too specific and detailed have sometime been factually inaccurate, while far too general wordings can be difficult to understand.
The focus of the assessments
Recommendations have focused on minimum standards
The descriptive sections of the evaluation reports generally present both the strengths and weaknesses of programmes. Good examples are often included. But the assessments and recommendations that are formulated on the basis of these descriptions have nearly always dealt with shortcomings in performance and proposed measures to rectify them. In this way the recommendations have involved setting a minimum level for these programmes and raising it. When the National Agency has decided to question entitlement to award degrees on the basis of the recommendations of a panel of assessors, a level has been set for the lowest acceptable educational standards.
More conditions and process than outcome
To provide a basis for the self-evaluations undertaken by the higher education institutions, a ‘manual for self-evaluation´ has been drawn up by the National Agency. This manual poses questions about both undergraduate and postgraduate programmes and these questions are grouped under three headings: conditions, process and outcomes. The questions probe different aspects of educational quality that can be deduced from the Higher Education Act and the Higher Education Ordinance and they have been chosen in consultation with the higher education institutions. The quality aspects included under these different headings have varied to some extent from year to year.[3]
The questions that indicated the outcomes of programmes in the guidelines for 2001 dealt with throughput, monitoring students, evaluation and quality assurance and attainment of objectives. The higher education institutions were asked to relate these outcomes to costs. In the case of postgraduate programmes there was also a question about the labour market for newly qualified PhD´s.
It is the responses to this relatively restricted list of questions on which the assessors have concentrated in evaluating the outcomes of the programmes. Considerably more questions were posed under the headings of “conditions" and “process". This means that on the whole relatively few judgements have been expressed that relate to what the programmes attain, i.e. the “outcomes" in the meaning given to the term in the National Agency´s manual.[4]
More attention paid to undergraduate programmes than to postgraduate courses
More judgements about undergraduate than about postgraduate programmes
All of the evaluations have normally contained recommendations that applied to both undergraduate and postgraduate programmes, for instance those dealing with national and international coordination, organisational conditions and teaching resources.
When recommendations have been addressed specifically to undergraduate or to postgraduate programmes, on the whole undergraduate programmes have been involved more often than those at postgraduate level. More judgements have been expressed and more recommendations made. In no evaluation has greater attention been paid to postgraduate programmes. The evaluations of mathematics and of economics devoted greater attention proportionally to postgraduate programmes than the other evaluations.
One reason why postgraduate programmes received less attention may be because they are smaller and perhaps easier to manage than at undergraduate level. It is possible that they are considered to have fewer and less significant problems. For example, in the evaluation of programmes in theology and religious studies postgraduate programmes appear to be relatively robust.
The recommendations are probably also governed by the delimitations in the National Agency´s evaluation mandate. The Agency and the panels of assessors are not required to express opinions about the research undertaken at the individual departments. In view of the close links between research and postgraduate programmes it is possible that this reduces the scope for comment. It may be difficult to formulate judgements and recommendations concerning postgraduate programmes without at the same time including research activities.
At undergraduate level there has moreover been clearer focus on the teaching itself than at postgraduate level. All the evaluations contain a relatively large number of assessments of the contents, structure and implementation of undergraduate programmes. In the context of postgraduate programmes the panels of assessors have frequently commented on the conditions that apply for doctoral students - funding, terms of employment, departmental duties and, to some extent, supervision - whereas on the whole they have been more cautious about opinions on the content and implementation of the programmes. However, relatively frequently the range of courses on offer was the subject of comment, then often in connection with a recommendation to collaborate.
As more recommendations were made, it is reasonable to assume that the national evaluation have, on the whole, had greater impact on undergraduate programmes in Sweden than at postgraduate level.
More opinions about the undergraduate programmes offered at university colleges than at universities
Somewhat more opinions have been expressed about the undergraduate programmes offered at the university colleges than at the universities. The evaluations have often contained more or less as many specific recommendations for each higher education institution, and as these apply only to undergraduate programmes at university colleges whereas for universities they include both undergraduate and postgraduate levels, the result is that fewer recommendations have been made about the undergraduate programmes at the universities than at the university colleges.
This could mean that the university colleges have been examined more closely than the universities, which would in its turn give the impression that there have been more problems in the undergraduate programmes at university colleges. In this case, however, there is no confirmation of this impression in the analyses of the panels of assessors. In no evaluation has the panel of assessors found that the standards in undergraduate programmes at university colleges were, in general, lower than those of the universities.
The contents and implementation of undergraduate programmes
Discussion of the content and organisation of every programme
In a few specific evaluations the panels of assessors have adopted a standpoint on what contents undergraduate programmes should have. This applies, for instance to the evaluations of media and communication studies, of economics, where the absence of certain courses could also be a reason for questioning entitlement to award degrees, and also in the evaluation of religious studies subjects and computer and systems science. These subjects vary in character. Two of them are new subjects in the process of development that have links with several other disciplines, one has a very broad disciplinary core and the fourth is a subject in which standards are both homogenous and well-established - in Sweden and internationally.
Generally, however, the discussions of the panels of assessors have rather concerned the principles on which the contents and organisation are based. Comments have been made in most of the evaluations on the range of courses offered, generally recommending breadth and at the same time consolidation at initial stages of programmes. At C (third semester) and D (fourth semester) levels the assessors have urged specialisation and the adoption of profiles in at least seven of the evaluations.
The higher education institutions have taken measures on the contents of undergraduate programmes
The opinions of the assessors on the contents and implementation of undergraduate programmes have seldom been challenged. On the other hand the panels´ recommendations have not always led to reactions.
The impression is that the recommendations that have mainly been followed are those that could be dealt with in the framework of ongoing course development procedures, in other words changes or modifications of the existing range of courses on offer. It has been less common to establish (many) entirely new courses or undertake more or less extensive reorganisation of the range of courses offered. In subjects like mathematics, economics and religious studies, for instance, the higher education institutions seem to have endeavoured generally to enhance the disciplinary profile that already existed, whereas more radical reallocation of priorities have been reported in only a few cases.
However, several new master´s programmes have been established, for instance, in economics, business administration and computer and system science.
The most frequent reason for not complying with recommendations about the contents of courses and the range offered is that the departments consider that they already provide the educational programme advocated by the panels of assessors.
Academic standards and research links figure in every evaluation
Links with research have been considered an important requirement for quality. Academic standards and links with research in undergraduate programmes are problems dealt with in all the evaluations and which attracted special emphasis in the evaluations of both economics, business administration, Latin/Greek/Modern Greek, Swedish/Nordic languages and religious studies.
At the same time these concepts appear to be vague. What the assessors pay attention to in their evaluations of research links and academic standards has varied. Most of them have emphasised the significant role played by teachers. All of the evaluations consider it important for teachers and those responsible for courses to have research qualifications. In a two evaluations the assessors advocate greater involvement by professors. It is also emphasised that those teaching courses should also devote time to their own research.
At the A (first semester) and B (second semester) levels it is also fairly common to find criticism of text books and other teaching materials and that the assessors perceive shortcomings in the teaching of methodology.
Discussion of the degree project courses at C and D levels can also be found in eight of the nine evaluations. Attention is drawn to shortcomings in the conditions in which students produce these projects, particularly supervision and the resources for supervision, and in several subjects the throughput is considered to be disturbingly low. The ways in which they are examined are also mentioned. The quality of the projects themselves, for instance the selection of problems and issues, links with theory and research, methodological and analytical level and their stringency, has only led to recommendations in the evaluation of linguistics and sign language.
The question of academic standards has also evoked reactions
The recommendations relating to academic standards and research links also seem to have fallen in good ground. To judge from the follow-up responses, the higher education institutions often agree with the observations of the assessors and during the three years that have elapsed since the evaluations took place, widespread efforts seem to have been devoted to the revision of reading lists. The teaching of methodology and the methods for supervising and examining degree projects seem to have been enhanced at some institutions but hardly overall in any subject.
Relatively fee recommendations about teaching and methods
The teaching in higher education is required to stimulate a scholarly approach. It is to foster critical thought, the students´ ability to solve problems and in-depth learning.
Analysis on the basis of these premises, however, can be found in few evaluations. The most frequent comments made by assessors about teaching in undergraduate programmes have more often related to concrete issues such as group size, examinations - the form they take, not their contents - and the need for methodological renewal, for instance through the use of modern media in teaching. One or two recommendations to specific institutions or of a more general nature in these areas are made in most of the evaluations. In the evaluations of business administration, economics, media and communication studies and Swedish/Nordic languages the reduction in the proportion of taught classroom hours is considered to jeopardise educational standards.
In several evaluations the issue of the inadequate grounding of newly enrolled students has been linked to the need for methodological development. In at least four evaluations the panels of assessors have come to the conclusion that the students´ prior knowledge of mathematics is far too poor, but shortcomings have been found in the prior knowledge of students in Swedish, statistics and programming.
Neither teaching nor methodological and educational issues have been given primary focus in any evaluation.
Mainly specific measures to develop methodology
The responses from the higher education institutions report only occasional examples of the renewal of teaching in undergraduate programmes. Certain departments have tried out new forms of instruction or teaching methods in response to the recommendations from the panels of assessors.
One area in which fairly general development seems to have taken place in programmes in Latin/Greek/Modern Greek and in business administration is the supervision of degree projects at the C and D levels. Resources for supervision have been augmented and a forum for supervisors established. In the programmes in economics there seems to have been a relatively general concentration on new forms of examination and several of the mathematics departments have tested new technological aids in their teaching.
The contents and implementation of postgraduate programmes
The evaluations have contained fewer recommendations for postgraduate programmes than for those at undergraduate level. Most of the recommendations for postgraduate programmes have concerned the range of courses offered, the terms on which postgraduate programmes are provided and the conditions offered to their students.
Focus on taught courses
Most of the evaluations contained specific recommendations that dealt with the taught courses provided in postgraduate programmes.
Sometimes the range of courses has been felt to be far too restricted; either because there were far too few courses or because the standard range of courses was felt to be far too limited. A relatively frequent recommendation was that postgraduate students should be offered courses in teaching methodology for higher education as part of their programmes. In the evaluation of Latin/Greek/Modern Greek the panel claimed that the reading load needed review and pruning as well coordination between the different departments and also that the balance between the thesis element and the taught courses should be reconsidered.
In several subjects the panels of assessors considered that taught courses should have stronger links with theses to enable throughput to be improved and make it easier for postgraduate students to complete their programmes in the stipulated time. The panel evaluating mathematics emphasised the importance of discussing the thesis subject early in the programme.
If postgraduate students had not already been allocated assistant supervisors, as a rule the panels of assessors have recommended that they should be. Otherwise few recommendations have been made with regard to supervision. Nor do seminar programmes seem to have given rise to many recommendations to specific higher education institutions.
The panels evaluating linguistics and sign language and computer and system science recommended the appointment of a director of studies for postgraduate programmes. This would help to create an overall approach to taught courses and supervision in postgraduate programmes.
Range of taught courses reviewed
As a rule recommendations that have concerned taught courses have been followed. Postgraduate students are more frequently offered the possibility of attending courses in teaching methodology for higher education and several seem to have been provided with access to an assistant supervisor. In Latin/Greek/Modern Greek the reading load has been pruned and coordinated. Some departments that did not receive specifically addressed recommendations of this type appear to have adopted the general recommendations.
Other recommendations concerning the implementation of postgraduate programmes have received a more varied response. Sometimes the higher education institutions have refrained from making any changes as they have wanted to wait for decisions to be taken in the ongoing Bologna process.
One or two of the follow-up responses state that individual study plans have become increasingly important tools in making postgraduate programmes more effective.
The recommended appointment of directors of postgraduate studies seems to have made a genuine impact in the institutions at which postgraduate programmes are offered in linguistics. In computer and systems science one department has appointed a director of studies.
Conditions vary for postgraduate programmes
The conditions in which postgraduate programmes are offered vary from subject to subject and also from higher education institution to higher education institution. The panel evaluating Latin/Greek/Modern Greek wrote that the 1998 reform of postgraduate programmes had had a devastating effect on small subjects that did not have access to external funding for their new postgraduate students. Seminar series were described as under threat. The panel also considered that more postgraduate studentships were required. Both the panel of assessors and the National Agency drew attention to the fact that at several higher education institutions postgraduate programmes did not even have access to any full-time teachers with advanced research qualifications.
In the evaluation of mathematics the panel of assessors stressed that new postgraduate programmes should not be launched unless certain fundamental criteria could be fulfilled, such guaranteeing that postgraduate students access to qualified supervision and a wide range of taught courses. Concentrating resources was recommended as was cooperation between different entities on the range of courses to be offered and supervision resources.
The panel of assessors evaluating media and communication studies pointed to the problems arising from dependence on external funding in research and postgraduate programmes and expressed the need for stable financial resources and staffing. The number of postgraduate studentships and post-doctoral posts was considered too small. In the evaluation of Swedish/Nordic languages the panel of assessors also stressed that more resources were needed for research and postgraduate programmes.
The panel of assessors evaluating economics referred to the difficulty of attaining critical mass as the subject is offered at so many higher education institutions, and in the evaluation of business administration the National Agency pointed out the importance of enabling the award of more PhD´s to meet the need for teachers with research qualifications. So that postgraduate teaching could take place in more creative and critical settings and to make better joint use of teaching and supervision resources most of the evaluations recommended greater national collaboration - sometimes within the framework of national graduate schools - as well as the adoption of profiles.
In the evaluation of Swedish/Nordic languages the National Agency emphasised the importance of cooperation between the major universities and the smaller university colleges so that students from institutions that did not offer postgraduate programmes could be appointed to postgraduate studentships.
The need to increase international exchange in postgraduate programmes was also pointed out in the evaluations of business administration, media and communication studies and economics.
Collaboration leads to improved conditions
The research environments in business administration have been enhanced. A rise in the number of new teachers and new postgraduate students has resulted from the evaluation of Latin/Greek/Modern Greek.
The national graduate school advocated by the National Agency for media and communication studies has not yet come into being but on the other hand a number of universities have combined to function as a network and have developed among other things a joint course in methodology for postgraduate programmes. A new postgraduate programme has been established by one university and internal graduate school at a university college, which means that both institutions now have a larger number of postgraduate students.
In mathematics a greater number of postgraduate students are participating in national graduate schools. National collaboration has also been augmented in economics and the conditions that apply to postgraduate programmes seem to have been improved in many quarters, for instance supervision resources have been expanded.
In Swedish /Nordic languages two of the major universities have reported that they now make appointments to postgraduate studentships on a competitive basis. However, they report that they see no value in adopting profiles as they consider it important to maintain the breadth of their postgraduate programmes.
Focus on conditions for postgraduate students in several evaluations
The conditions for postgraduate students are referred to in most of the evaluations, for instance in urging that the same conditions should apply both to those with external funding and internal funding. Another recommendation that can be found in several evaluations is that postgraduate students should be credited with more working hours for the preparation of their teaching. In media and communication studies, for instance, the assessors could see that teaching in undergraduate courses was largely offered by postgraduate students and that their departmental duties could be extensive enough to hinder their continued postgraduate studies. In the case of business administration the National Agency pointed out a certain lack of compliance with the Higher Education Ordinance at two independent university colleges with regard to the teaching required of postgraduate students and also in the use of individual study plans.
In a number of evaluations the assessors recommended that postgraduate students should be given teaching in courses that had links to their thesis subjects and that they could act as assistant supervisors for degree projects at the C-level.
Some improvement in conditions for postgraduate students
The higher education institutions specifically recommended to review and regulate the time spent by postgraduate students on teaching seem, on the whole, to have remedied the problems. In business administration, for instance, the postgraduate students´ departmental duties have been reduced in several quarters. It also seems that more postgraduate students are being offered teaching in areas relating to their thesis topics. The recommendation that postgraduate students could also act as assistant supervisors for C-level degree projects does seem to have gained any major acceptance.
In the cases where recommendations concerning postgraduate students´ conditions have not led to any action, this has often been for economic reasons. The higher education institutions have not always been able to afford to replace grant funding for the initial years of postgraduate study with postgraduate studentships.
All evaluations raise issues relating to teachers
Lack of teaching resources in every subject
The situation for teaching staff has been a central theme in all of the evaluations, the issue which on the whole has been given greatest emphasis. The panels of assessors have made general recommendations as well as recommendations to specific higher education institutions about teachers´ qualifications, both in terms of the overall ratio of teachers with PhD´s or other research qualifications as well as the proportion of teachers with research qualifications in undergraduate programmes. In all the evaluations the panels of assessors and the National Agency have considered that teaching resources in undergraduate programmes have in some respects been far too meagre. Six of the total of eight instances in which entitlement to award degrees was questioned and nine of the eleven grave reprimands were based at least in part on the view that teaching resources were inadequate.
In the evaluations of computer and systems science and of linguistics and sign language the teachers were described as qualified but far too few in number.
The panels of assessors have primarily recommended new recruitments, but they have also mentioned promotion for senior lecturers and professors and enabling lecturers to add to their qualifications, as well as further measures to raise the number of postgraduate degrees awarded in order to raise the number of teachers with research qualifications.
Many initiatives to raise the numbers of teachers and their qualifications
The attention paid to the predicament of the teachers in the evaluations seems to have been more than justified. In the follow-up responses most higher education institutions report at least attempts, aims or hopes of improving the situation. The recommendations that teaching resources should be enhanced have not been challenged.
Just over three years after the evaluations, it also seems that manifest changes have taken place in a number of subjects, for instance business administration, Latin/Greek/Modern Greek and media and communication studies. More teachers have been appointed, but above all the academic qualifications of the teaching staff seem to have been raised.
On the basis of other responses it is more difficult to say to what extent the situation has in fact been improved. In all subjects the appointment of some new teachers is reported. At the same time, at many higher education institutions resources have been invested to improve the qualifications of lecturers and senior lecturers, and teachers with research qualifications have been promoted. On the other hand, it is not always clear what impact the newly appointed teachers or the enhancement of their qualifications has had for specific departments or subjects - if they have led to any decisive changes for the better in relation to the needs that existed.
The most common reason why new appointments have not been made or no scope has been allowed for the enhancement of academic qualifications is lack of economic resources, despite the intentions that existed. The shortage of qualified applicants has also played a role in some subjects.
There has been a radical reduction in the number of applications for programmes in computer science in recent years and therefore the need for more and better qualified teachers has declined.
Problematisation of teachers´ working conditions
The conditions in which teachers teach and do research have been problematised. In all the evaluations the restricted scope for teachers to undertake their own research or enhance their qualifications in some other way has been one of the issues raised. The evaluations show that in most higher education institutions time had been allocated for teachers to do research within their posts but that teaching and administration took up most of their working hours and also tended to encroach on research. The scope for research was also affected by the way in which the time available was organised.
The overall workloads of the teachers were also considered to be demanding, above all in Latin/Greek/Modern Greek, mathematics and media and communication studies. Not infrequently, the panels of assessors have regarded them as far too demanding and in a few subjects at least they seem to have increased in recent years.
Few measures adopted to create scope for research and reduce pressure
Judging from the follow-up responses, there has been no general increase in the scope available for teachers to undertake their own research or enhance their qualifications in any other way since the evaluations took place. One or two positive examples are reported from individual higher education institutions, and they often involve individual teachers or groups of teachers, but the main impression is that only modest improvements have taken place.
Virtually no examples can be found of any reduction of teachers´ workloads.
Gender bias in most subjects
In seven of the evaluations the panels of assessors commented on the gender bias in the subjects and urged measures to deal with it. The large proportion of male teachers and postgraduate students was problematised, often in relation to the preponderance of women among students.
In the follow-up responses individual measures are reported that affect the gender ratio.
The focus on collaboration has made an impact
National collaboration recommended in all evaluations
The need for national collaboration has been a recurrent theme in all of the evaluations. The panels of assessors have advocated collaboration primarily as a way of using limited national resources more efficiently. Collaboration is presented as a way of improving the supply of teachers at individual higher education institutions and to extend the range of courses offered at C and D levels and in postgraduate programmes. In their evaluations of the postgraduate programmes in computer and systems science, Latin/Greek/Modern Greek, media and communications studies and economics the panels also advocated the establishment of new graduate schools or greater participation by individual departments in the graduate schools that already existed.
Another way of reallocating national resources is through concentration. This requires either the collaboration of a number of higher education institutions that agree on some mutual allocation of responsibilities for aspects of the teaching in the subject or a decision by the authorities on national allocation of tasks and resources. This is a recommendation that has only been addressed explicitly to the mathematics departments -concentration of their resources to environments with soundly established postgraduate programmes.
Individual higher education institutions have increased collaboration
The follow-up responses contain descriptions of how individual collaborative activities have been augmented and how new partners have identified each other. The impression is that on the whole national collaboration has increased in all or nearly all subjects.
It does not appear to be equally as common for the evaluations to have helped to launch more extensive national collaboration of any greater significance for the various subjects. One positive example is, however, the new collaborative groupings in economics that seem to have changed the conditions that apply to postgraduate programmes in many quarters. Another example is the new subject conferences that Sweden´s departments of Swedish have agreed to arrange. Relatively extensive cooperation has been arranged between the programmes in religious studies to make it easier to transfer credits from one to the other and to move between them. Cooperation between programmes in Latin/Greek/Modern Greek also seems to have grown considerably.
International collaboration recommended in most reports
In six of nine evaluations - the exceptions being linguistics/sign language, Latin/Greek/Modern Greek and mathematics - the panels of assessors considered that internationalisation in the subject was far too modest. Their recommendations generally involved stimulating teacher and student exchanges.
Other methods proposed included the review of reading lists, offering more courses in English and enhancement of the linguistic skills of teachers.
Some departments have increased the number exchanges
The follow-up responses mainly describe individual measures by individual departments. No more widespread measures to boost international exchanges seem to have been adopted in any subject.
However, a relatively large number of new master´s programmes seem to have been established in business administration. In media and communication studies as well, many departments report augmentation of their student exchanges. It has turned out to be more difficult to extend the number of teacher exchanges or place them on a more formal basis.
Internal organisation and division of responsibilities
Divisions and lack of support in certain quarters
In six of the evaluations organisational conditions became an important issue. Two types of problems emerged. When core subjects are linked to a number of different disciplines, as in business administration, mathematics, media and communication studies and programmes in religious studies and theology, departments not infrequently found it difficult to establish subject coherence or a common approach among teachers, and internal organisational arrangements could at times exacerbate these problems. The size of a subject, in terms of the student population and number of teaching staff, could also play a role. For subjects with greater geographic dispersion, above all business administration and economics, often with limited numbers in “multidisciplinary departments", the assessors could see problems linked to lack of internal support within the higher education institutions and also sometimes the way in which the subject was administered.
In the evaluation of Latin/Greek/Modern Greek the panel of assessors were also able to identify “a number of serious disagreements between individuals".
Greater integration than strengthened positions
The follow-up responses suggest that it has been easier for the higher education institutions to deal with problems relating to the integration of the different elements of programmes than to strengthen the position of a subject in the higher education institutions themselves. Media and communication studies and also religious studies are subjects in which greater integration is reported, but several business administration departments seem to have made no little progress.
The lack of reports from the higher education institutions of more improvements in administration and internal support may be related to the fact that changes of this kind are not always visible at departmental level. A few departments claim as well that circumstances were not as problematic as the panel of assessors suggested, or perceived them to be.
The overall organisational conditions at two university colleges were criticised by three different panels of assessors. At both there has been radical reorganisation.
Quality assurance is important - sometimes
Major variations between different panels of assessors
The panels of assessors have viewed quality assurance issues in different ways. In some evaluations (computer and system science, linguistics, media and communication studies, Swedish/Nordic languages) many recommendations are made to individual departments about their quality assurance. Others (business administration, Latin/Greek/Modern Greek) make extremely few. In most of the evaluations the National Agency has used more stringent wording than the panels of assessors with regard to the quality assurance procedures at the higher education institutions.
Some panels of assessors have noted the need to formulate objectives and consider strategies, while another regarded lack of student influence as a relatively major issue. In the evaluation of media and communication studies both the National Agency and the panel of assessors considered that the lack of systematic quality assurance was remarkable. Quality assurance systems are also referred to by the panel evaluating Swedish/Nordic languages. In the evaluation of linguistics and sign language the panel of assessors stresses the importance of student participation in planning programmes, for example in drawing up syllabuses.
The panels of assessors evaluating Latin/Greek/Modern Greek and economics also expressed a desire for some system to monitor alumni.
The only quality assurance issue raised more generally dealt with course evaluations and feed-back of their results.
Quality assurance procedures have been affected in various ways
The follow-up responses show that in the subjects in which many recommendations were made to specific higher education institutions, a great many measures have also been adopted. The general recommendations seem to have been disregarded to a large extent. One exception is in mathematics where routines for course evaluations have been improved considerably even through extremely few specific recommendations were addressed to any departments. One possible explanation is that the draconian wording used by the National Agency stating that “particular attention will be paid to the systems at the higher education institutions for course evaluation" has had an impact.
The most common measures adopted to improve quality assurance procedures involve course evaluation routines. The recommendations to formulate objectives and ensure support for them have not had the same impact. Monitoring alumni has started at some higher education institutions in some subjects but hardly to the extent recommended in the evaluations. Student influence seems to have been enhanced generally in at least three of the subjects evaluated.
In this chapter an initial analysis has been presented of the more general significance of the national evaluations for quality assurance procedures and development work at the higher education institutions.
[1] One of the evaluations contained no recommendations addressed to specific higher education institutions. It only contains the “Panel of Assessors´ comments".
[2] Some of the decisions attached by the Swedish National Agency for Higher Education to the evaluations of subjects and programmes contained what were described as “grave reprimands" instead of or as well as calling entitlement to award degrees into question. These involved a less serious degree of criticism and were to be reviewed within one year. The “grave observations" system was abandoned from 2002 and onwards.
[3] The quality aspects on which the National Agency based evaluations in 2001 were: recruitment and composition of student/postgraduate student populations; teachers´ qualifications and scope for skill enhancement: the aims, contents and organisation of programmes; libraries and the provision of other information; premises and equipment; study situation for students/postgraduate students; teachers´ workloads; types of examination; a critical and creative educational setting; evaluation and quality assurance; monitoring results and throughput. One quality aspect that has been added in more recent years is diversity. Questions dealing with distance teaching and flexible forms of instruction have also been added.
[4] The manifestly greater space devoted to conditions and implementation than to outcomes is related both to the way in which the concepts are defined in the evaluations and the overall difficulties that exist in defining educational outcomes.
Using the conditions, process, outcomes structure reflects an endeavour to provide a holistic presentation of the quality of programmes. It is assumed that their organisation and implementation is governed by the conditions in which they offered and that it should therefore be possible to derive outcomes from both conditions and the process. The idea is that everything is interrelated. In practice there is no fixed point at which conditions terminate and the process takes over or where the process finishes and the outcomes can be seen. The boundaries between what should be considered a “condition", “process" or “outcome" are relative.
The reason why the National Agency has nevertheless opted to group the questions/quality aspects under different headings is for the sake of clarity. The three headings underline the holistic perspective and at the same time lay down a joint structure for the self-evaluations.
Other difficulties related to the definition of educational outcomes are for instance distinguishing between outcomes that can be ascribed directly to a programme and the kind of outcomes that are related to the individual characteristics of the students or with conditions in the world around them.
Content responsible: , e-mail: forename.surname@hsv.se