The recommendations also seem to have tallied with the adaptations to the Bologna process taking place at undergraduate and postgraduate level in many places.
What has been asserted above all about the evaluations involving the major subjects and subjects that are either relatively new or have multidisciplinary features, such as economics, business administration, computer and systems science as well as media and communication studies, is how useful it has been to be able to make comparisons with other departments. It is difficult for teachers and researchers who work with subjects that are spread across many departments to acquire a national overview, while for others who are involved in study programmes that are in the process of being developed and whose contents involve links with other disciplines external inspiration is valuable. “The assessors saw a few risks in the way we worked and in our relationship to long-term funding that we should probably have already been taking more seriously at the time, which shows the value of the outsider´s bird´s eye view in evaluations like these" is how this was summarised by one of the media and communication studies programmes that took part in the evaluation.
Some individual higher education institutions express a desire for more international comparisons. During a period when adaptations are being made to the Bologna process these can valuable.
The evaluations have also been used to improve conditions in individual higher education institutions, as is shown by the follow-up of the evaluations of Latin/Greek/Modern Greek, mathematics, economics as well as Swedish and Nordic languages. The evaluation results have provided new arguments and greater weight in internal discussions about the allocation of resources to the subject, for example, or its organisational parameters.
Several institutions claim, however, it can sometimes be difficult to know if the changes described in the follow-up responses really are the results of evaluation. It is likely that the changes would have been made even without the intervention of the National Agency. The national evaluations have involved a scrutiny that has in its turn prompted, accelerated or provided a basis for measures that would in some cases have been adopted sooner or later in any case.
In a few follow-up responses, a total of around 20, the question is raised of whether the benefits of the evaluations correspond to the cost and the effort that they have involved.
The follow-up suggests that the higher education institutions have primarily adopted recommendations that were addressed both generally and in the specific analyses.
The follow-up shows, however, that not only the recommendations have been followed. Many of the other assessments of the panels and the National Agency that were not expressed as recommendations seem to have prompted action. The higher education institutions have not always made a distinction between assessment and recommendation.
In every case the doubts expressed about entitlement to award degrees were withdrawn after one year and no entitlement was revoked. On the other hand a few programmes were terminated after the evaluation. At Göteborg University Modern Greek was no longer offered as an independent subject.
The higher education institutions seem principally to have adopted the relatively simple proposals, i.e. those that it has been comparatively inexpensive to implement and which have not required a great deal of effort from the teachers and other members of the staff. The responses account primarily for measures that it has been possible to adopt within the framework of normal operational development.
The reactions to other types of recommendations have been more varied. Changes that require action by participants who do not form part of the subject setting or the department itself have not always come about. In some evaluations, for instance, the organisational situation of a subject within a specific higher education institution has been questioned and the measures proposed by the panel of assessors have not infrequently affected other departments or faculties and required action by the institution´s central administrators. In these cases it has not been unusual for more weight to be given to internal considerations and regulations than to the National Agency´s evaluations, one reason being that these merely take into account the conditions that apply for individual subjects.
Nor has action always been the outcome of recommendations that require the involvement of participants from outside the higher education institutions themselves, mainly other higher education institutions. In all the evaluations, for instance, the panels of assessors and the National Agency have advocated greater efforts to attain national coordination. These recommendations seem mainly to have resulted in the initiation or enhancement by specific institutions of their collaboration with other individual institutions, not infrequently on the initiative of individuals or groups. More extensive coordination arrangements that encompass several higher education institutions and have been of general significance for the subjects have not come into being as often. There has not been sufficient incentive to stimulate widespread joint endeavours. The exhortations of the National Agency and of the panels of assessors have not carried enough weight.
Another reason why certain recommendations have not been followed is that the higher education institutions have not considered they could afford them. Recommendations that require a great deal of expenditure seem rather to have been perceived as provocative at a time when the institutions found themselves in straitened economic circumstances.
Some higher education institutions have refrained from adopting measures as they have opted to wait for decisions to be made in connection with the ongoing Bologna process, for instance about the new degree ordinance.
Finally, the way in which recommendations have been worded has also played its role. Value judgements that were far too specific and detailed have sometime been factually inaccurate, while far too general wordings can be difficult to understand.
The questions that indicated the outcomes of programmes in the guidelines for 2001 dealt with throughput, monitoring students, evaluation and quality assurance and attainment of objectives. The higher education institutions were asked to relate these outcomes to costs. In the case of postgraduate programmes there was also a question about the labour market for newly qualified PhD´s.
It is the responses to this relatively restricted list of questions on which the assessors have concentrated in evaluating the outcomes of the programmes. Considerably more questions were posed under the headings of “conditions" and “process". This means that on the whole relatively few judgements have been expressed that relate to what the programmes attain, i.e. the “outcomes" in the meaning given to the term in the National Agency´s manual.
When recommendations have been addressed specifically to undergraduate or to postgraduate programmes, on the whole undergraduate programmes have been involved more often than those at postgraduate level. More judgements have been expressed and more recommendations made. In no evaluation has greater attention been paid to postgraduate programmes. The evaluations of mathematics and of economics devoted greater attention proportionally to postgraduate programmes than the other evaluations.
One reason why postgraduate programmes received less attention may be because they are smaller and perhaps easier to manage than at undergraduate level. It is possible that they are considered to have fewer and less significant problems. For example, in the evaluation of programmes in theology and religious studies postgraduate programmes appear to be relatively robust.
The recommendations are probably also governed by the delimitations in the National Agency´s evaluation mandate. The Agency and the panels of assessors are not required to express opinions about the research undertaken at the individual departments. In view of the close links between research and postgraduate programmes it is possible that this reduces the scope for comment. It may be difficult to formulate judgements and recommendations concerning postgraduate programmes without at the same time including research activities.
At undergraduate level there has moreover been clearer focus on the teaching itself than at postgraduate level. All the evaluations contain a relatively large number of assessments of the contents, structure and implementation of undergraduate programmes. In the context of postgraduate programmes the panels of assessors have frequently commented on the conditions that apply for doctoral students - funding, terms of employment, departmental duties and, to some extent, supervision - whereas on the whole they have been more cautious about opinions on the content and implementation of the programmes. However, relatively frequently the range of courses on offer was the subject of comment, then often in connection with a recommendation to collaborate.
As more recommendations were made, it is reasonable to assume that the national evaluation have, on the whole, had greater impact on undergraduate programmes in Sweden than at postgraduate level.
More opinions about the undergraduate programmes offered at university colleges than at universities
Somewhat more opinions have been expressed about the undergraduate programmes offered at the university colleges than at the universities. The evaluations have often contained more or less as many specific recommendations for each higher education institution, and as these apply only to undergraduate programmes at university colleges whereas for universities they include both undergraduate and postgraduate levels, the result is that fewer recommendations have been made about the undergraduate programmes at the universities than at the university colleges.
This could mean that the university colleges have been examined more closely than the universities, which would in its turn give the impression that there have been more problems in the undergraduate programmes at university colleges. In this case, however, there is no confirmation of this impression in the analyses of the panels of assessors. In no evaluation has the panel of assessors found that the standards in undergraduate programmes at university colleges were, in general, lower than those of the universities.
Generally, however, the discussions of the panels of assessors have rather concerned the principles on which the contents and organisation are based. Comments have been made in most of the evaluations on the range of courses offered, generally recommending breadth and at the same time consolidation at initial stages of programmes. At C (third semester) and D (fourth semester) levels the assessors have urged specialisation and the adoption of profiles in at least seven of the evaluations.
The higher education institutions have taken measures on the contents of undergraduate programmes
The opinions of the assessors on the contents and implementation of undergraduate programmes have seldom been challenged. On the other hand the panels´ recommendations have not always led to reactions.
The impression is that the recommendations that have mainly been followed are those that could be dealt with in the framework of ongoing course development procedures, in other words changes or modifications of the existing range of courses on offer. It has been less common to establish (many) entirely new courses or undertake more or less extensive reorganisation of the range of courses offered. In subjects like mathematics, economics and religious studies, for instance, the higher education institutions seem to have endeavoured generally to enhance the disciplinary profile that already existed, whereas more radical reallocation of priorities have been reported in only a few cases.
However, several new master´s programmes have been established, for instance, in economics, business administration and computer and system science.
The most frequent reason for not complying with recommendations about the contents of courses and the range offered is that the departments consider that they already provide the educational programme advocated by the panels of assessors.
At the same time these concepts appear to be vague. What the assessors pay attention to in their evaluations of research links and academic standards has varied. Most of them have emphasised the significant role played by teachers. All of the evaluations consider it important for teachers and those responsible for courses to have research qualifications. In a two evaluations the assessors advocate greater involvement by professors. It is also emphasised that those teaching courses should also devote time to their own research.
At the A (first semester) and B (second semester) levels it is also fairly common to find criticism of text books and other teaching materials and that the assessors perceive shortcomings in the teaching of methodology.
Discussion of the degree project courses at C and D levels can also be found in eight of the nine evaluations. Attention is drawn to shortcomings in the conditions in which students produce these projects, particularly supervision and the resources for supervision, and in several subjects the throughput is considered to be disturbingly low. The ways in which they are examined are also mentioned. The quality of the projects themselves, for instance the selection of problems and issues, links with theory and research, methodological and analytical level and their stringency, has only led to recommendations in the evaluation of linguistics and sign language.
Analysis on the basis of these premises, however, can be found in few evaluations. The most frequent comments made by assessors about teaching in undergraduate programmes have more often related to concrete issues such as group size, examinations - the form they take, not their contents - and the need for methodological renewal, for instance through the use of modern media in teaching. One or two recommendations to specific institutions or of a more general nature in these areas are made in most of the evaluations. In the evaluations of business administration, economics, media and communication studies and Swedish/Nordic languages the reduction in the proportion of taught classroom hours is considered to jeopardise educational standards.
In several evaluations the issue of the inadequate grounding of newly enrolled students has been linked to the need for methodological development. In at least four evaluations the panels of assessors have come to the conclusion that the students´ prior knowledge of mathematics is far too poor, but shortcomings have been found in the prior knowledge of students in Swedish, statistics and programming.
Neither teaching nor methodological and educational issues have been given primary focus in any evaluation.
One area in which fairly general development seems to have taken place in programmes in Latin/Greek/Modern Greek and in business administration is the supervision of degree projects at the C and D levels. Resources for supervision have been augmented and a forum for supervisors established. In the programmes in economics there seems to have been a relatively general concentration on new forms of examination and several of the mathematics departments have tested new technological aids in their teaching.
Sometimes the range of courses has been felt to be far too restricted; either because there were far too few courses or because the standard range of courses was felt to be far too limited. A relatively frequent recommendation was that postgraduate students should be offered courses in teaching methodology for higher education as part of their programmes. In the evaluation of Latin/Greek/Modern Greek the panel claimed that the reading load needed review and pruning as well coordination between the different departments and also that the balance between the thesis element and the taught courses should be reconsidered.
In several subjects the panels of assessors considered that taught courses should have stronger links with theses to enable throughput to be improved and make it easier for postgraduate students to complete their programmes in the stipulated time. The panel evaluating mathematics emphasised the importance of discussing the thesis subject early in the programme.
If postgraduate students had not already been allocated assistant supervisors, as a rule the panels of assessors have recommended that they should be. Otherwise few recommendations have been made with regard to supervision. Nor do seminar programmes seem to have given rise to many recommendations to specific higher education institutions.
The panels evaluating linguistics and sign language and computer and system science recommended the appointment of a director of studies for postgraduate programmes. This would help to create an overall approach to taught courses and supervision in postgraduate programmes.
One or two of the follow-up responses state that individual study plans have become increasingly important tools in making postgraduate programmes more effective.
The recommended appointment of directors of postgraduate studies seems to have made a genuine impact in the institutions at which postgraduate programmes are offered in linguistics. In computer and systems science one department has appointed a director of studies.
In the evaluation of mathematics the panel of assessors stressed that new postgraduate programmes should not be launched unless certain fundamental criteria could be fulfilled, such guaranteeing that postgraduate students access to qualified supervision and a wide range of taught courses. Concentrating resources was recommended as was cooperation between different entities on the range of courses to be offered and supervision resources.
The panel of assessors evaluating media and communication studies pointed to the problems arising from dependence on external funding in research and postgraduate programmes and expressed the need for stable financial resources and staffing. The number of postgraduate studentships and post-doctoral posts was considered too small. In the evaluation of Swedish/Nordic languages the panel of assessors also stressed that more resources were needed for research and postgraduate programmes.
The panel of assessors evaluating economics referred to the difficulty of attaining critical mass as the subject is offered at so many higher education institutions, and in the evaluation of business administration the National Agency pointed out the importance of enabling the award of more PhD´s to meet the need for teachers with research qualifications. So that postgraduate teaching could take place in more creative and critical settings and to make better joint use of teaching and supervision resources most of the evaluations recommended greater national collaboration - sometimes within the framework of national graduate schools - as well as the adoption of profiles.
In the evaluation of Swedish/Nordic languages the National Agency emphasised the importance of cooperation between the major universities and the smaller university colleges so that students from institutions that did not offer postgraduate programmes could be appointed to postgraduate studentships.
The need to increase international exchange in postgraduate programmes was also pointed out in the evaluations of business administration, media and communication studies and economics.
The national graduate school advocated by the National Agency for media and communication studies has not yet come into being but on the other hand a number of universities have combined to function as a network and have developed among other things a joint course in methodology for postgraduate programmes. A new postgraduate programme has been established by one university and internal graduate school at a university college, which means that both institutions now have a larger number of postgraduate students.
In mathematics a greater number of postgraduate students are participating in national graduate schools. National collaboration has also been augmented in economics and the conditions that apply to postgraduate programmes seem to have been improved in many quarters, for instance supervision resources have been expanded.
In Swedish /Nordic languages two of the major universities have reported that they now make appointments to postgraduate studentships on a competitive basis. However, they report that they see no value in adopting profiles as they consider it important to maintain the breadth of their postgraduate programmes.
In a number of evaluations the assessors recommended that postgraduate students should be given teaching in courses that had links to their thesis subjects and that they could act as assistant supervisors for degree projects at the C-level.
In the cases where recommendations concerning postgraduate students´ conditions have not led to any action, this has often been for economic reasons. The higher education institutions have not always been able to afford to replace grant funding for the initial years of postgraduate study with postgraduate studentships.
In the evaluations of computer and systems science and of linguistics and sign language the teachers were described as qualified but far too few in number.
The panels of assessors have primarily recommended new recruitments, but they have also mentioned promotion for senior lecturers and professors and enabling lecturers to add to their qualifications, as well as further measures to raise the number of postgraduate degrees awarded in order to raise the number of teachers with research qualifications.
Just over three years after the evaluations, it also seems that manifest changes have taken place in a number of subjects, for instance business administration, Latin/Greek/Modern Greek and media and communication studies. More teachers have been appointed, but above all the academic qualifications of the teaching staff seem to have been raised.
On the basis of other responses it is more difficult to say to what extent the situation has in fact been improved. In all subjects the appointment of some new teachers is reported. At the same time, at many higher education institutions resources have been invested to improve the qualifications of lecturers and senior lecturers, and teachers with research qualifications have been promoted. On the other hand, it is not always clear what impact the newly appointed teachers or the enhancement of their qualifications has had for specific departments or subjects - if they have led to any decisive changes for the better in relation to the needs that existed.
The most common reason why new appointments have not been made or no scope has been allowed for the enhancement of academic qualifications is lack of economic resources, despite the intentions that existed. The shortage of qualified applicants has also played a role in some subjects.
There has been a radical reduction in the number of applications for programmes in computer science in recent years and therefore the need for more and better qualified teachers has declined.
The overall workloads of the teachers were also considered to be demanding, above all in Latin/Greek/Modern Greek, mathematics and media and communication studies. Not infrequently, the panels of assessors have regarded them as far too demanding and in a few subjects at least they seem to have increased in recent years.
Virtually no examples can be found of any reduction of teachers´ workloads.
In the follow-up responses individual measures are reported that affect the gender ratio.
Another way of reallocating national resources is through concentration. This requires either the collaboration of a number of higher education institutions that agree on some mutual allocation of responsibilities for aspects of the teaching in the subject or a decision by the authorities on national allocation of tasks and resources. This is a recommendation that has only been addressed explicitly to the mathematics departments -concentration of their resources to environments with soundly established postgraduate programmes.
It does not appear to be equally as common for the evaluations to have helped to launch more extensive national collaboration of any greater significance for the various subjects. One positive example is, however, the new collaborative groupings in economics that seem to have changed the conditions that apply to postgraduate programmes in many quarters. Another example is the new subject conferences that Sweden´s departments of Swedish have agreed to arrange. Relatively extensive cooperation has been arranged between the programmes in religious studies to make it easier to transfer credits from one to the other and to move between them. Cooperation between programmes in Latin/Greek/Modern Greek also seems to have grown considerably.
Other methods proposed included the review of reading lists, offering more courses in English and enhancement of the linguistic skills of teachers.
However, a relatively large number of new master´s programmes seem to have been established in business administration. In media and communication studies as well, many departments report augmentation of their student exchanges. It has turned out to be more difficult to extend the number of teacher exchanges or place them on a more formal basis.
In the evaluation of Latin/Greek/Modern Greek the panel of assessors were also able to identify “a number of serious disagreements between individuals".
The lack of reports from the higher education institutions of more improvements in administration and internal support may be related to the fact that changes of this kind are not always visible at departmental level. A few departments claim as well that circumstances were not as problematic as the panel of assessors suggested, or perceived them to be.
The overall organisational conditions at two university colleges were criticised by three different panels of assessors. At both there has been radical reorganisation.
Some panels of assessors have noted the need to formulate objectives and consider strategies, while another regarded lack of student influence as a relatively major issue. In the evaluation of media and communication studies both the National Agency and the panel of assessors considered that the lack of systematic quality assurance was remarkable. Quality assurance systems are also referred to by the panel evaluating Swedish/Nordic languages. In the evaluation of linguistics and sign language the panel of assessors stresses the importance of student participation in planning programmes, for example in drawing up syllabuses.
The panels of assessors evaluating Latin/Greek/Modern Greek and economics also expressed a desire for some system to monitor alumni.
The only quality assurance issue raised more generally dealt with course evaluations and feed-back of their results.
The most common measures adopted to improve quality assurance procedures involve course evaluation routines. The recommendations to formulate objectives and ensure support for them have not had the same impact. Monitoring alumni has started at some higher education institutions in some subjects but hardly to the extent recommended in the evaluations. Student influence seems to have been enhanced generally in at least three of the subjects evaluated.
In this chapter an initial analysis has been presented of the more general significance of the national evaluations for quality assurance procedures and development work at the higher education institutions.
 One of the evaluations contained no recommendations addressed to specific higher education institutions. It only contains the “Panel of Assessors´ comments".
 Some of the decisions attached by the Swedish National Agency for Higher Education to the evaluations of subjects and programmes contained what were described as “grave reprimands" instead of or as well as calling entitlement to award degrees into question. These involved a less serious degree of criticism and were to be reviewed within one year. The “grave observations" system was abandoned from 2002 and onwards.
 The quality aspects on which the National Agency based evaluations in 2001 were: recruitment and composition of student/postgraduate student populations; teachers´ qualifications and scope for skill enhancement: the aims, contents and organisation of programmes; libraries and the provision of other information; premises and equipment; study situation for students/postgraduate students; teachers´ workloads; types of examination; a critical and creative educational setting; evaluation and quality assurance; monitoring results and throughput. One quality aspect that has been added in more recent years is diversity. Questions dealing with distance teaching and flexible forms of instruction have also been added.
 The manifestly greater space devoted to conditions and implementation than to outcomes is related both to the way in which the concepts are defined in the evaluations and the overall difficulties that exist in defining educational outcomes.
Using the conditions, process, outcomes structure reflects an endeavour to provide a holistic presentation of the quality of programmes. It is assumed that their organisation and implementation is governed by the conditions in which they offered and that it should therefore be possible to derive outcomes from both conditions and the process. The idea is that everything is interrelated. In practice there is no fixed point at which conditions terminate and the process takes over or where the process finishes and the outcomes can be seen. The boundaries between what should be considered a “condition", “process" or “outcome" are relative.
The reason why the National Agency has nevertheless opted to group the questions/quality aspects under different headings is for the sake of clarity. The three headings underline the holistic perspective and at the same time lay down a joint structure for the self-evaluations.
Other difficulties related to the definition of educational outcomes are for instance distinguishing between outcomes that can be ascribed directly to a programme and the kind of outcomes that are related to the individual characteristics of the students or with conditions in the world around them.