Master-class October 2018: Meta-Analysis

This master class introduces and applies recently developed methods of meta-analysis to integrate, summarize and understand rapidly expanding research in the health, business and the social sciences.  We demonstrate how these methods address and mitigate the sources of the current credibility crisis across the disciplines.


This 2 day course is being held in ACSPRI's Melbourne office.



Monday, October 8, 2018 - Tuesday, October 9, 2018
Early bird cutoff date: 
Monday, September 17, 2018
Course details:

This master class introduces and applies recently developed methods of meta-analysis to integrate, summarize and understand rapidly expanding research in the health, business and the social sciences.  We demonstrate how these methods address and mitigate the sources of the current credibility crisis across the disciplines.



This course will be co-taught by Dr Emily Kothe & Professor Tom Stanley:


Dr Emily Kothe is the Co-Director of DeLMAR and the Stream Leader for Systematic Review and Meta-Analysis in the School of Psychology Data Sciences Unit. She has conducted systematic reviews and meta-analyses across a number of topics within psychology including: attachment, cognitive dissonance, health behaviour and behaviour change, and body image.

Master Class - runs over 2 days

Tom Stanley is Professor of Meta-Analysis at the School of Business, Deakin University. He has published many papers developing and applying methods for meta-regression and to reduce publication bias. He is the Associate Editor of the Journal of Economic Surveys and the convener of MAER-Net (Meta-Analysis of Economics Research-Network). Both Tom and Emily are members of DeLMAR (Deakin Lab for the Meta-Analysis of Research).

Course dates: Monday 8 October 2018 - Tuesday 9 October 2018
Course status: Course completed (no new applicants)
Week 1
About this course: 

Selective reporting, questionable research practices, low statistical power, and wide variation of findings across an area of research are the causes of the current credibility/replication crisis.  Meta-analysis and meta-regression analysis can identify the reach of these problems in your area of research, filter out much of their effects, and thereby reduce their influence.  When carefully conducted, meta-analysis can greatly increase research credibility.  


The advanced methods presented in this course are designed to address these issues directly.  Meta-regression analysis routinely explains much of the wide and often conflicting variation typically found among reported findings. Recently developed meta-analysis methods use a study’s statistical power to reduce these potential biases.  We aim to provide participants with the tools needed to produce state-of-the-art research synthesis and meta-analyses.  


The workshop is relevant to all researchers in health, medicine, psychology, economics, business and other social sciences could benefit from this class if they are already somewhat knowledgeable about meta-analysis.

Course syllabus: 

Day 1

1) The Current Replication/Credibility ‘Crisis’ (John et al., 2012; Ioannidis et al., 2017)
    a) Publication bias, questionable research practices and heterogeneity
    b) What meta-analyses tell us about contemporary research.
        i) Replicability (OSC, 2015; Stanley et al., 2017)
        ii) Statistical power (Ioannidis et al., 2017; Stanley et al., 2017)
        iii) Publication bias
        iv) Heterogeneity


2) Review of Meta-Analysis (MA) (Hunter & Schmidt, 2000, Borenstein et al., 2009)
    a) Effect sizes
    b) Fixed and Random Effects
    c) Application to psychology and health research
    d) Issues and Limitations


3) Methods to Accommodate and Reduce Publication Bias (Stanley, 2008)
    a) Causes: Questionable Research Practices (Ferguson & Heene,2012; Franco et al.,2014)
    b) Funnel graphs and motivating examples (Stanley and Doucouliagos, 2010)
    c) Unrestricted Weighted Least Squares (WLS) (Stanley and Doucouliagos, 2015)
    d) FAT-PET-PEESE and other methods (Stanley and Doucouliagos, 2014)



Day 2

1) High Heterogeneity among Research Results (Higgins & Thompson, 2002)
    a) How high heterogeneity limits research synthesis
    b) How prevalent is it? (Ioannidis et al, 2017; Stanley et al, 2017)
    c) Meta-analysis methods to measure and accommodate heterogeneity


2)  Introduction to Meta-Regression Analysis (MRA) (Borenstein et al., 2009)
    a) Basic MRA models and their interpretation
    b) Applications from psychology, economics and health


3) MRA: Advanced Topics with extensive applications (Stanley and Doucouliagos, 2012)
    a) Robustness of MRA findings
    b) Seeing through the fog of Multicollinearity
        i) General-to-specific modelling
        ii) Bayesian Model Averaging
    c) Dependence across reported estimates (Hedges et al., 2010)
        i) Cluster-robust standard errors
        ii) Fixed and random-effects panel MRA
       iii) Multilevel models (Cheung, 2014).


4) ‘Hands on’ lab: Using R  and STATA for MA and MRA



Course format: 

This two day master-class will be held in the ACSPRI Office in Melbourne.


Attendees should bring a laptop with either (a) STATA or (b) R and RStudio installed. Instructions with how to install R and RStudio will be circulated before the workshop. If you don’t have a copy of Stata, could you please contact us in advance & we will organise a trial version for you for the course.

Recommended Background: 

Participants should be conducting a meta-analysis or have attended a previous meta-analysis workshop or class. Participants also need to be knowledgeable about basic research statistics and regression analysis.

Recommended Texts: 

There are no required readings for this class, although before attending, we recommend reading:

  • Doucouliagos, C. (2016) Meta-regression analysis: Producing credible estimates from diverse evidence. IZA World of Labor: 320. doi:10.15185/izawol.320.


Further References:

  • Borenstein, M., Hedges, L. V., Higgins, J. P. T. and Rothstein, H. R. (2009). Introduction to meta-analysis, Chichester, U.K. John Wiley & Sons.
  • Cheung, M. W.-L. (2014). Modeling dependent effect sizes with three-level meta-analyses: A structural equation modeling approach. Psychological Methods, 19(2), 211-229.
  • Franco, A., Malhotra, N., & Simonovits, G. (2014). Publication bias in the social sciences: Unlocking the file drawer. Science, 345(6203), 1502-1505.
  • Ferguson, C. J., & Heene, M. (2012). A vast graveyard of undead theories: Publication bias and psychological science’s aversion to the null. Perspectives on Psychological Science, 7(6),555-561.
  • Hedges, L. V., Tipton, E., & Johnson, M. C. (2010). Robust variance estimation in meta‐regression with dependent effect size estimates. Research Synthesis Methods, 1(1), 39-65.
  • Higgins, J., & Thompson, S. G. (2002). Quantifying heterogeneity in a meta‐analysis. Statistics in Medicine, 21(11), 1539-1558.
  • Hunter, J. E., & Schmidt, F. L. (2000). Fixed effects vs. random effects meta‐analysis models: Implications for cumulative research knowledge. International Journal of Selection and Assessment, 8(4), 275-292.
  • Ioannidis, J.P.A., Stanley, T.D. and Doucouliagos, C. (2017). “The power of bias in economics research,” The Economic Journal, 127: F236-265.
  • John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524-532.
  • Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716–aac4716. doi:10.1126/science.aac4716
  • Stanley, T.D. (2008). Meta-regression methods for detecting and estimating empirical effect in the presence of publication selection. Oxford Bulletin of Economics and Statistics, 70:103-27.
  • Stanley, T. D., Carter, E. C., & Doucouliagos, H. (2017). What meta-analyses reveal about the replicability of psychological research. Unpublished manuscript.
  • Stanley, T.D and Doucouliagos, H. (2010). Picture this: A simple graph that reveals much ado about research.” Journal of Economic Surveys, 24: 170-91.
  • Stanley, T.D. and Doucouliagos, H. (2012). Meta-regression analysis in economics and business. Oxford: Routledge.
  • Stanley, T. D., & Doucouliagos, H. (2014). Meta‐regression approximations to reduce publication selection bias. Research Synthesis Methods, 5(1), 60-78.
  • Stanley, T.D. and Doucouliagos, C. “Neither fixed nor random: Weighted least squares meta-analysis,” Statistics in Medicine 34 (2015), 2116-27.
  • Stanley, T.D., Doucouliagos, C. and Ioannidis, J.P.A. (2017). “Finding the Power to Reduce Publication Bias,” Statistics in Medicine, 36: 1580-1598.
Course fees
Non Member: 
Full time student Member: 

The ACSPRI Head Office is located at 763 Heidelberg Road, Alphington, just before the corner of Yarralea St.



The ACSPRI Office is only 20 minutes from the Melbourne's CBD via train. Accommodation located close to any of the City Loop train stations (Parliament, Melbourne Central, Spencer St, Flinders St and {on weekdays} Flagstaff) or other stations along the Hurstbridge Line will enable a very easy commute (see Transport below). We expect most visitors would appreciate staying quite centrally and there are numerous accommodation options. It is worth checking a travel website (expedia, tripadvisor, etc) to fine the best option for your budget and requirements, but we recommend staying close to a train station as per above.


Ivanhoe Quest Apartments are located 20 minutes walk east of the ACSPRI Office, along Upper Heidelberg Road.  You can also catch the train to Alphington Station from either Ivanhoe Station (2 stops - city direction) or Darebin Station (1 stop - city direction). You can also walk to Heidelberg Road (4 mins) take the 546 Bus to Yarralea Street, which is opposite our office.



Car Parking / Transport: 

Alphington Station is roughly 20 minutes from Melbourne Central (and other City loop stations) on the Hurstbridge Line and only a few minutes' walk from the ACSPRI office. There are also a number of bus lines that service the area. Bus services are very limited on weekends and the train is recommended. Click here for the journey planner.


By car, the quickest route for many local Melbournians will be via the Eastern Freeway and/or Chandler Hwy. From the CBD the Eastern Freeway from the Inner North will also probably be the quickest route. At that time on a weekend 20 minutes would allow plenty of time. Parking is usually available in Yarralea St or nearby but not on Heidelberg Road. 


Secure space is available to store bicycles at the office. The office is very close to Alphington Park, which backs onto the Yarra River and subsequently a vast network of Melbourne's bike paths.



Venue and Timetable: 

The course will run from 9.30am until 4.30pm each day (unless otherwise notified).

Breaks are usually held at 11:00 - 11:15, 12:45 - 1.15 and 2:45 - 3:00. Lunch and a small morning and afternoon tea are provided each day.