In medical sciences (particularly, epidemiological and evidence-based medicine), meta-analysis is an accepted method. It is said that famous statisticians like Karl Pearson (to overcome power of reduced statistical power in small sample studies, 1904), RA Fisher (idea of cumulating probability values because the insingnificant results of a number of indipendent studies may give a lower picture from probabilty then one would obtain by chance, 1944), WG Cochran (in his discussion on averaging of means across independent studies laid down the statistical foundation through inverse variance weighting and homogenity testing) among others contributed towards this (see, Meta-analysis in Wikipedia and Practical Meta-Analysis by David B Wilson).
It was Gene V. Glass who first made use of the term in its statistical sense in a seminal paer Primary, secondary, and meta-analysis of research, Educational Researcher, 5 (10), 3-8, 1976, JSTOR link. To quote from this paper:
"Primary analysis is the original analysis of data in a research study. It is what one typically imagines as the application of statistical methods."
"Secondary analysis is the re-analysis of data for the purpose of answering the original research question with better statistical techniques, or answering new questions with old data."
"Meta-analysis refers to ... the statistical analysis of a large collection of analysis results from individual studies for the purpose of integrating the findings. It connotes a rigorous alternative to the casual, narrative discussions of research studies which typify our attempts to make sense of the rapidly expanding research literature."
I have been aware of this technique because of my research interests in the public health domain. But, what caught my attention very recently the paper by Derek D. Headey and Andrew Hodge, 'The Effect of Population Growth on Economic Growth: A Meta-Regression Analysis of the Macroeconomic Literature' Population and Development Review, 35 (2), 221-248, 2009. The study explores how differences in the method could account for differences in results.
The meta-regression technique is as follows. In the first step, one has to rely on the independent original studies on a particular theme. In these studies the regressions will have a dependent variable (like economic growth, Yi) and an independent variable (like population growth, Xi) and there may be a set of control variables (Z), which are likely to be different for different sutdies.
(1) Yi=a0+a1Xi+ajZj+ei (j>1)
To know the impact of population growth on economic growth, the relevant coefficient is a1. Its statistical significance depending on its standard error, s1; that is, if the t-stat=(a1/s1)>|2|.
In the second step, the main task for the meta-regression, one the t-stat of the k studies (t1k) on the (el, l) number of methodological dummies.
(2) tk=b0+blDl+uk
The methodological dummies could refer to alternative measures of economic growth (Gross Domestic Product (GDP) per capita, Gross National Product (GNP) per capita, GDP per worker), alternative measures of population growth (total population, young population, working population) or because of alternative techniques used (control set includes health indicator, sample is developing land-poor countries, sample is developing countries, sample is for a prticular time period, the econometric method uses weighted least squares and so on and so forth). Of course, the more number of dumies one can use would depend upon the more number of studies that already exist.
Some of the problems with meta-analysis identified in literature are the following.
The studies that one is likely to have access (read published) are the ones that would have reported significant results. In other words, unpublished studies that showed negative or null findings are difficult to find.
Another problem that comes from critiques of the approach is th 'Apples versus Oranges'. Is it proper to compare the different studies together. But, the independent studies are also comparing something. It is true that one should be careful and know what one is comparing than doing away with comparision altogether.
There is also the 'Flat Earth' criticism. That an analysis of averages may not do justice. Yes, and one should use meta-analysis judiciously to explore that also (for instance, in the use of methodological dummies). There are many others that one would come across, as one reads on more.
A good read is Meta-analysis at 25 by none other than Gene V Glass. Researchers in other disciplines should explore this method. Go head on!
No comments:
Post a Comment