School of Design (SD) SFQ form

The new Faculty/School-based SFQ forms have two major sections:

The Educational Development Centre (EDC) is responsible for co-ordinating the implementation of the Faculty/School-based SFQ, analysing the numerical data from the SFQ, and generating the SFQ report which shows the summary of statistical results for each staff member.

The SFQ results will be one of the contributory elements of the PolyU staff appraisal system, and should thus be used for both developmental and judgmental purposes. However, it should be noted that the ratings indicate only the perceptions of the students taking the subject. They are only one of the sources of information concerning a staff member's contribution to teaching. Individual staff members are encouraged to bring forward other materials or evidence of their contribution to teaching (e.g. peer review results or teaching portfolios) for discussion in staff appraisal meetings, or in support of their applications for contract renewals, promotions, or awards on the basis of their merits in teaching. Please refer to the PolyU guidelines on teaching evaluation by following this link.

Understanding the SFQ Report

Essentially, the SFQ report shows a summary of the statistical results of the responses provided by the specific class of students to the various SFQ items on their learning experience of the subject and the teaching of the staff member. It has three main sections:

Information about the staff, subject, and administration

The first part of the report shows the background information about the staff member, the subject, and the administration details, including

• the name and staff number of the staff member concerned, his/her employment status and departmental affiliation,
• the programme code, subject name and code, and group number if applicable,
• other pertinent background information about the subject/teaching, including:

- the programme and subject levels,
- the mode and nature of the subject,
- the language of instruction,
- the semester when the SFQ was administered,
- the number of students enrolled, the number of completed SFQ forms returned, and the response rate, and
- the focus of feedback and the part of teaching being evaluated.

It should be noted that for this sort of student feedback surveys, a response rate of 70% or above is normally expected, particularly for smaller classes (e.g. classes with an enrolment under 20). If the number of returns or the response rate is too low, the results should be interpreted with great caution.

Back to top

About the subject

The second part of the report shows the summary statistics for the first four standard closed-response type items on students’ learning experience of the subject, and the percentage distribution of students’ response to the fifth item on subject workload.

For the first four standard items on subject learning experience, the mean and standard deviation of the students’ ratings are reported. Moreover, the percentage distribution of the students' responses choosing “strongly agree”, “agree” and “no strong view” is plotted graphically on the right hand side of the item statistics.

For these four items, the item scores range from 1 to 5, where a rating of 1 means strongly disagree, a rating of 3 means no strong view, and a rating of 5 means strongly agree. A low mean score (i.e. a score closer to 1) on any item implies that students generally disagree with the statement of the item, while a high mean score (i.e. a score closer to 5) implies that students generally agree with it.

The standard deviation shows the degree of variability of the students' responses to that particular item. A high standard deviation means that the students vary widely in their opinions, and a low standard deviation means that there is a high level of agreement among students in their views on the particular item.

The diagram on the right of the item score gives a graphical representation of the percentage distribution of the students' ratings in the “strongly agree”, “agree” and “no strong view” categories. A concentration of students' responses in the shaded portion of the bar (i.e. response categories of “agree” and “strongly agree”) signifies that most students tend to agree with that item. On the other hand, the “gap” between the bar and the vertical axis on the right represents the proportion of students choosing “disagree” or “strongly disagree”. The larger the gap, the higher is the proportion of students disagreeing with the item.

For the standard item on subject workload, it is not appropriate to compute the mean score and standard deviation. Thus, only a graphical representation of the percentage distribution of students’ responses in each of the response categories (i.e. too heavy, appropriate or too light) is shown.

About the staff member

The last part of the report shows the summary statistics for the Faculty/School-based items on the teaching of the staff member, the two university-wide standard items on the overall view of the staff member’s teaching and their grand means, as well as the percentage distribution of students’ responses to the item on the use of English (or Putonghua for some specific classes) in teaching.

For the Faculty/School-based items on the teaching of the staff member and the two university-wide standard items on students’ overall view, the mean and standard deviation of the students’ ratings are reported. Moreover, the percentage distribution of the students' responses choosing “strongly agree”, “agree” and “no strong view” is plotted graphically on the right hand side of the item statistics.

For these items, the item scores range from 1 to 5, where a rating of 1 means strongly disagree, a rating of 3 means no strong view, and a rating of 5 means strongly agree. A low mean score (i.e. a score closer to 1) on any item implies that students generally disagree with the statement of the item, while a high mean score (i.e. a score closer to 5) implies that students generally agree with it.

The standard deviation shows the degree of variability of the students' responses to that particular item. A high standard deviation means that the students vary widely in their opinions, and a low standard deviation means that there is a high level of agreement among students in their views on the particular item.

The diagram on the right of the item score gives a graphical representation of the percentage distribution of the students' ratings in the “strongly agree”, “agree” and “no strong view” categories. A concentration of students' responses in the shaded portion of the bar (i.e. response categories of “agree” and “strongly agree”) signifies that most students tend to agree with that item. On the other hand, the “gap” between the bar and the vertical axis on the right represents the proportion of students choosing “disagree” or “strongly disagree”. The larger the gap, the higher is the proportion of students disagreeing with the item.

For the item on the use of English (or Putonghua) in teaching, a table showing the percentage distribution of students’ responses in each of the response categories (i.e. all or nearly all of the time, majority of the time, about half or less than half of the time) and the percentage of missing data is included, as the data are not amenable for the computation of mean scores and standard deviation.

Open-ended questions As students' responses to the standardised open-ended questions are not susceptible to statistical analysis, they will not be shown in the SFQ report. The SFQ forms completed by students will be returned to the staff member concerned for his/her information via the respective department/school office, after the finalisation of the examination results at Exam Board Meetings.

Back to top

Guidelines for Interpreting the SFQ Results

It should be remembered that the ratings shown in the report indicate only the general reactions of the students to the subject/teaching. They are not absolute or precise measures of the teaching effectiveness of the teacher/teaching, and thus should never be viewed as such. The results need to be interpreted with great caution, as some variations in the results across teachers can be expected because of measurement errors and/or factors that are beyond their control.

The following guidelines may be helpful when interpreting SFQ results:

• Student feedback should be interpreted in context: the teaching context must be considered when feedback from a particular group of students is reviewed. Contextual factors may include class size, level and year of study, nature of the subject (core vs. elective, theoretical vs. applied, ...), nature of the teaching format (lecture vs. tutorial vs. practical or clinical sessions, ... ), etc. Such factors are often beyond the control of the staff member but nonetheless would influence the feedback of a particular class.
• The numbers and figures should not be seen as an absolute measure of the teaching performance of the staff member. Small differences in the student ratings may not have any statistical or practical significance at all.
• SFQ feedback is only one source of evidence for judging teaching performance. Other forms of feedback such as students' comments, staff-student discussions, External Examiner's Report, peer review results, etc. should also be taken into account.
• SFQ results are best regarded as a rough indicator of students' experience of learning rather than a precise and objective measure of the teaching performance of a staff member. A low rating on a particular scale or item signals the need for further investigation rather than a hasty judgment and action. Very often, improvements require co-ordinated effort of several staff; and changing the context may be just as important as changing the behaviour of the individual staff member.
• As classes taught by individual staff members are different, it is not very useful to crudely compare the ratings of one staff member with those of another without due recognition of their differences in context. For developmental purposes, the most appropriate way of using student ratings is to track a staff member's ratings over time, and to identify aspects which are causing increasing dissatisfaction or concern to students so that improvements can be made at appropriate times.
• When passing evaluative judgment on the teaching performance of the staff member concerned in relation to important personnel decisions, it is more appropriate to consider student feedback from multiple classes collected over a period of time, and to base the judgment on the basis of a wider range of evidence included in a teaching portfolio as recommended in the PolyU guideline on teaching evaluation.

Using the Faculty/School-based SFQ Norms

What is a 'norm'?

To put it simply, a norm is a set of average ratings derived by including in its computation a large number of cases in a specified reference group. The Faculty/School-based SFQ norms are given in a separate document entitled Cumulative Norms for SFQ Scores by Faculty/School. The norms are derived by taking the average of a large number of class-sets of student ratings on the SFQ collected over a sustained period of time. Separate norms are also developed for different comparison groups from different Faculties/Schools with different class sizes. The norms are revised regularly to provide updated information about the average ratings of the various reference groups.

Purposes of having norms

Norms are useful in two ways. First, they provide a reference group for relative comparisons; that is, how the ratings of a particular class compare to the ratings of other classes in the reference group. This may help the staff member to identify the relative strengths and weaknesses of his/her teaching, and areas for improvement. Second, they help the interpretation of possible biases in ratings caused by factors beyond the control of the staff member. With the development of separate norms for different teaching contexts (for example, class size), the influence of the factors that define the norm groups can be taken into account.

However, it must be stressed that the primary purpose of these normative comparisons is not to pass judgment on staff members but rather, to give them a sense of their relative performance; for self reflection and development. As we will see below, there are problems with using normative comparisons and so, extra caution should be taken in interpreting the results.

Considerations in normative comparisons

Normative comparisons are problematic because by the very nature of norms, about half of the staff members are below average in any reference group. This means that even a highly effective staff member may have a relatively lower standing if other staff members in the reference group are rated highly by the students. In other words, a lower than average rating does not necessarily mean that the staff member is ineffective or incompetent.

Furthermore, changes or improvements in ratings over time are difficult to interpret because there will be a corresponding change in the norms at the same time.

The appropriateness of the reference group used for comparison is another problem. Students' ratings are often affected by factors such as class size, level of study, and nature of the subject, etc. which are outside the control of the staff member. It would be futile to compare ratings between staff members teaching in completely different contexts.

The following points should be considered when making normative comparisons:

• Avoid treating the norms as an absolute standard. Norms should never be viewed as a line of demarcation between pass and fail in teaching performance.
• It is normal to expect variations in students’ ratings among staff members. Unless the ratings are significantly higher or lower than the averages (for example, when the score is more than 1 standard deviation above or below the norm, or lies below the 10^th or above the 90^th percentile scores of the norm), they may not have any statistical or practical significance.
• Avoid using normative comparisons as the sole factor for judgment. Other sources of information such as students’ open-ended comments, test results, staff-student discussions, peer review results, etc. should also be taken into account.
• It is more useful to use normative comparisons to reveal the relative strengths and weaknesses of the individual staff members, and to identify possible areas for improvement.
• Choose an appropriate norm for comparison. As far as possible, compare the ratings of a specific class-set with the norm or reference group with matching class size, level of study, or nature of subject.

Understanding the tables of norms

Each table portrayed in the document entitled Cumulative Norms for SFQ Scores by Faculty/School shows the means, standard deviations, the minimum and maximum scores for each of the standardised close-response type items on the Faculty/School-based SFQ of the class-sets included as the reference group in its computation. It also shows the 10^th, 25^th, 50^th, 75^th and 90^th percentile scores for each of the items, and the total number of class-sets of data included in the computation.

To establish the relative standing of the ratings of a staff member, select the most appropriate table of norms, compare each of the item scores of the staff member with the respective mean of the item in the table, and examine the extent of deviation (positive or negative) from the mean. Ratings significantly higher than the mean score in the table signifies the relative strengths of the staff member's teaching, while ratings significantly lower than the mean in the table suggest relative weaknesses.

Another way to identify the relative strengths and weaknesses of a staff member's teaching is to compare his/her ratings of the SFQ items with the corresponding percentile scores in the table. A rating below the 10^th percentile score of the norm implies that the staff member's rating is among the bottom 10% of the classes in the reference group on that item. On the other hand, if a staff member's rating is above the 90^th percentile score, it means that he/she ranks at the top 10% of the classes in the reference group on that item. Through these comparisons, it will be possible to establish the relative standing of the staff member's teaching for each of the SFQ items as compared to the reference group.

Back to top

What to Do Next for Improving Teaching?

Collecting student feedback is the very first step in improving teaching. However, the feedback results will not automatically lead to improvement unless the staff members concerned follow up with plans for development or improvement. This may involve an investigation into students’ problems and concerns, a systematic reflection on ones' teaching, discussions and sharing of experiences and insights with other colleagues, and development of an action plan for improvement. Very often, improvements require co-ordinated effort of several staff, and require changes in the curriculum and learning systems as well as the behaviour of the individual staff members. The support and encouragement from the department is critical in such attempts.

How to make use of the SFQ results to improve teaching?

Teachers may wish to consider the following steps, in planning teaching improvements.

1. Interpreting the SFQ scores
Study the SFQ report and compare the ratings with the appropriate norms to identify:

• the relative strengths and weaknesses of the teaching, and
• the aspects of teaching which are causing most concern to your students.

2. Finding out more about the students' views

Examine the students’ responses to the open-ended questions in order to understand more about:

• the aspects of the teaching that the students found most helpful to their learning,
• what changes in the teaching the students think may help them learn better, and
• other comments or suggestions made by the students.

3. Developing an action plan for improvement
While the teaching per se may affect students' learning and generate some of their comments, other concerns may be the result of the design of courses, or subjects or other factors that are beyond the classroom teaching. Some of these factors may be outside the teacher’s control. Discussions with colleagues and students may help to clarify some of these factors, and suggest plans for development. Again, teachers may find it helpful to discuss their plan with colleagues in the department, or any EDC staff member.

4. Implementing the plan

It is important to monitor the effects of any changes made by collecting students’ (and colleagues’) feedback. As the changes made may involve or affect other members of the department, it is important to talk with them about the plans, and their outcomes.

EDC will also be happy to help or assist in any way that is useful.

How the EDC can help

EDC is given the responsibility to support and assist PolyU staff in teaching improvements. Colleagues may ask for the following EDC services if they need them:

• help in the interpretation of the SFQ results,
• assistance in other forms of collecting student feedback or evaluation,
• advice and consultation on teaching, curriculum design and learning system design,
• help in producing learning resources (print-based or multimedia) to support teaching and learning, and
• advice and consultation on action learning projects on teaching improvements, etc.

In addition, EDC offers a number of short courses, workshops and seminars on university teaching. It also has a collection of practical books, monographs, and reference materials on teaching in university. These may provide useful ideas for teaching. Please approach any educational development officer in EDC for information or assistance.

EDC Contact Persons

Please contact the persons indicated below for assistance in interpreting SFQ results, planning teaching improvements based on the feedback, or devising alternative forms of evaluation.

For	Contact
Obtaining SFQ forms Returning SFQ forms for data analysis	Miss Jenny Yung, EO (x6292) E-mail: etjyung
Enquiries concerning SFQ procedures and guidelines Enquiries concerning data analysis Advice on interpreting SFQ results	Dr. K.P. Kwan, SEDO (x6320) E-mail: etkpkwan
Advice on other forms of evaluation Advice on teaching improvements based on SFQ results	Dr. Angela Ho, Head (x6282) E-mail: etangela Dr. K.P. Kwan, SEDO (x6320) E-mail: etkpkwan
Comments and suggestions on the procedures and instruments	Dr. K.P. Kwan, SEDO (x6320) E-mail: etkpkwan