B-school rankings: Setting the expectations
Abhishek Agrawal May 29, 2014In my previous column, I underlined the importance of ranking professional educational institutes.
During the ranking exercise, a comparison of institutes is carried out, within a certain geography. on several parameters of relevance and their sub-parameters. The critical issue is: whose inputs to take for these comparisons? Common sense says only those who have availed the services of B-school programmes have the knowhow on what should be compared and what should not.
If one strictly goes by the "experience" factor, it is impossible to find a person who has experienced all or several B-schools. A person generally experiences business management education at one of the many B-schools. However, during that experience he/ she develops enough ideas and insights on what makes a B-school better than another by way of discussions during admission processes, college-fests, inter-college competitions, during summer internships and at the time of placement. He/ she can say with certain amount of confidence that institute X is better than institute Y among a set of institutes.
It is important to compare a service or product from the perspectives of demand as well as supply. The supply side should be given an opportunity to present its credentials, while the demand side (users) would be best to judge these service providers based on their experience of the outcome. Hence BT-MDRA has devised the ranking based on three key components:
A. Objective data:
B. Experience-based Perceptual data:
a. Academia - deans/ directors (peer ranking)
The BT-MDRA methodology does not limit itself to getting the data from right set of respondents, but also ensures the accuracy of data. The validation and audit of the data is done religiously.
With this methodology wherein the inputs are taken from demand and supply - both sides, based on experience, that is further validated through a systematic process; it aims to make itself the most credible ranking
However, like any exercise it has certain limitations, such as:
a. Measurement: Not every parameter can be measured. As someone truly said, "Beauty lies in the beholder's eyes", but beauty contests are held across the globe every year. Similarly, even if some parameters cannot be measured, a comparison is important based on measurable differences. However, every year we aim to make more attributes measurable and try to use them too without affecting the comparability of rankings year-on-year.
b. Participation: To have a fair comparison, the honest participation of the institutes is important. Not all are motivated to participate due to various reasons - some obvious some non-obvious. In the absence of participation by certain B-schools due to whatsoever reasons, the entire ranking table has a chance to go topsy-turvy.
c. Relativity: Like any competition or comparison, ranking is a relative exercise. The evaluation and scoring are relative to each other among the participating institutes. Therefore by non-participation of some B-schools, the entire community loses out in two ways - i) by not knowing about the non-participated B-school and ii) relative rankings of other B-schools going up or down.
Despite these limitations, the BT-MDRA ranking approach tries to be most comprehensive in terms of taking the facts and viewpoints based on experience. It considers the inputs of widest set of stakeholders from the supply and demand sides of the B-school spectrum. It also does not mix up the 1-year programs with 2-year programs as is done by several rankings.
However, there is no single ranking either in India or globally that has been left out of criticism. Most of the rankings published by Indian magazines take perception-based data. Even if some rankings use objective data, they lack the rigour in validation of data.
Talking about the rankings by world's reputed publications; each one of them has its own merits and demerits. The six most widely referred global rankings are briefly discussed below:
50-30-20 Rule: The weighted average of the previous year's data is also considered for the current ranking although the percentage decreases with year.
Pros: It features regional ranking. The 50-30-20 Rule makes in reducing volatility.
2. Forbes: It is based on return-on-investment concept. Forbes sends out surveys to MBA alumni five years after graduation and asks their current salary and bonus. The magazine then subtracts out two years of forgone pre-MBA pay, as well as tuition and fees-minus scholarship aid. The actual Forbes ranking is based on the net cumulative amount alumni of a school have earned. The magazine adjusts the median five-year MBA gain for cost of living expenses and discounts gains using a rate tied to money market yields.
Pros: Much more effective for price sensitive students/stakeholders
3. Financial Times (FT): Alumni responses on eight key criteria together (salary data contributing highest) contribute 59per cent weight. Eleven criteria (measuring diversity and international reach) are calculated from school data, accounting for 31per cent of total weight. A research rank accounting for 10per cent weight based on number of articles published by full-time faculty in certain journals.
Pros: Multi dimensional- includes "value for money" perspective as well. When taking alumni ratings, considers 3-year ratings where available.
4. Bloomberg Business Week: Judges B-schools on how well they serve their two main constituencies: students and corporate recruiters. Based on the web-based survey rating responses of students and recruiters. Students give ratings on 10-point scale on teaching quality, career services, and other aspects. Recruiters rate the programs based on perceived quality of grads and company's experience with MBAs.
Pros: Uses three most recent student surveys to ensure that short term issues do not skew results. Focus on customer satisfaction.
5. QS Global: The QS Global 200 Business Schools survey captures the preferred set of business schools each responding employer wishes to recruit. Each time a school is selected by an employer, it receives one vote and the total number of votes is referred to as the 'total unprompted votes.' Employers are then asked to identify the schools they regard as attractive for the purpose of hiring MBA graduates Each time a school is voted, it is given one vote, referred to as the 'total prompted votes'. The prompted and unprompted votes are added together to create the 'total employer votes'. Ranking is done according to 'index of employer votes'.
Pros: Ranks according to employers' perspective which helps in analyze the current demands and trends.
6. US News: It uses a wide variety of data sources. Quality assessment (weight 0.40) is based on peer assessment score (rating by other B-schools deans and directors, weight 0.25) and recruiter assessment score (survey among corporate recruiters, weight 0.15). Placement success (weight 0.35) includes mean starting salary and employment rates of grads. Student selectivity (weight 0.25) includes mean GMAT and GRE scores, mean undergraduate GPA and acceptance rate.
Pros: Multi-Dimensional approach based on quality assessment, placement success and student selectivity.
Therefore each of the rankings methodology has their own advantages and shortcomings. Nonetheless, rankings are quite useful in several aspects of decision making. We at BT-MDRA try to evolve our approach year-on-year by incorporating new measurable elements and learning from our past mistakes to make it most credible and most sought-after rankings. By far, it is also most transparent in terms of publishing the details.
(The author is Executive Director, MDRA)