Alexander Peysakhovich
ABSTRACT
Scientific and business practices are increasingly resulting in large collections of randomized experiments. Analyzed together multiple experiments can tell us things that individual experiments cannot. We study how to learn causal relationships between variables from the kinds of collections faced by modern data scientists: the number of experiments is large, many experiments have very small effects, and the analyst lacks metadata (e.g., descriptions of the interventions). We use experimental groups as instrumental variables (IV) and show that a standard method (two-stage least squares) is biased even when the number of experiments is infinite. We show how a sparsity-inducing l0 regularization can (in a reversal of the standard bias--variance tradeoff) reduce bias (and thus error) of interventional predictions. We are interested in estimating causal effects, rather than just predicting outcomes, so we also propose a modified cross-validation procedure (IVCV) to feasibly select the regularization parameter. We show, using a trick from Monte Carlo sampling, that IVCV can be done using summary statistics instead of raw data. This makes our full procedure simple to use in many real-world applications.
- Joshua D Angrist, Guido W Imbens, and Donald B Rubin. 1996. Identification of causal effects using instrumental variables. J. Amer. Statist. Assoc. Vol. 91, 434 (1996), 444--455.Google Scholar
Cross Ref
- Joshua D Angrist and Alan B Krueger. 1995. Split-sample instrumental variables estimates of the return to schooling. Journal of Business & Economic Statistics Vol. 13, 2 (1995), 225--235.Google Scholar
- Joshua D Angrist and Jörn-Steffen Pischke. 2008. Mostly Harmless Econometrics: An Empiricist's Companion. Princeton university press.Google Scholar
- Susan Athey and Guido Imbens. 2016. Recursive partitioning for heterogeneous causal effects. Proceedings of the National Academy of Sciences, Vol. 113, 27 (2016), 7353--7360.Google ScholarCross Ref
- E. Bakshy, D. Eckles, and M. S. Bernstein. 2014. Designing and Deploying Online Field Experiments. Proceedings of the 23rd ACM conference on the World Wide Web. ACM. Google ScholarDigital Library
- Abhijit Banerjee and Esther Duflo. 2012. Poor Economics: A Radical Rethinking of the Way to Fight Global Poverty. PublicAffairs.Google Scholar
- Paul A Bekker. 1994. Alternative approximations to the distributions of instrumental variable estimators. Econometrica: Journal of the Econometric Society (1994), 657--681.Google Scholar
- Alexandre Belloni, Daniel Chen, Victor Chernozhukov, and Christian Hansen. 2012. Sparse models and methods for optimal instruments with an application to eminent domain. Econometrica, Vol. 80, 6 (2012), 2369--2429.Google ScholarCross Ref
- Léon Bottou. 2014. From machine learning to machine reasoning. Machine Learning, Vol. 94, 2 (2014), 133--149. Google ScholarDigital Library
- Léon Bottou, Jonas Peters, Joaquin Quinonero Candela, Denis Xavier Charles, Max Chickering, Elon Portugaly, Dipankar Ray, Patrice Y Simard, and Ed Snelson. 2013. Counterfactual reasoning and learning systems: The example of computational advertising. Journal of Machine Learning Research Vol. 14, 1 (2013), 3207--3260. Google ScholarDigital Library
- Bob Carpenter, Andrew Gelman, Matt Hoffman, Daniel Lee, Ben Goodrich, Michael Betancourt, Michael A Brubaker, Jiqiang Guo, Peter Li, and Allen Riddell. 2016. Stan: A probabilistic programming language. Journal of Statistical Software (2016).Google Scholar
- Gary Chamberlain and Guido Imbens. 2004. Random effects estimators with many instrumental variables. Econometrica, Vol. 72, 1 (2004), 295--306.Google ScholarCross Ref
- Dean Eckles, René F Kizilcec, and Eytan Bakshy. 2016. Estimating peer effects in networks with peer encouragement designs. Proceedings of the National Academy of Sciences, Vol. 113, 27 (2016), 7316--7322.Google ScholarCross Ref
- Ziv Epstein, Alexander Peysakhovich, and David G Rand. 2016. The good, the bad, and the unflinchingly selfish: Cooperative decision-making can be predicted with high accuracy when using only three behavioral types Proceedings of the 2016 ACM Conference on Economics and Computation. ACM, 547--559. Google ScholarDigital Library
- John C Gittins. 1979. Bandit processes and dynamic allocation indices. Journal of the Royal Statistical Society. Series B (Methodological) (1979), 148--177.Google Scholar
- Mathew Goldman and Justin M Rao. 2014. Experiments as Instruments: Heterogeneous Position Effects in Sponsored Search Auctions. Available at SSRN 2524688 (2014).Google Scholar
- Donald P Green, Shang E Ha, and John G Bullock. 2010. Enough already about “black box” experiments: Studying mediation is more difficult than most scholars suppose. The Annals of the American Academy of Political and Social Science, Vol. 628, 1 (2010), 200--208.Google ScholarCross Ref
- Justin Grimmer, Solomon Messing, and Sean J Westwood. 2014. Estimating heterogeneous treatment effects and the effects of heterogeneous treatments with ensemble methods. Unpublished manuscript, Stanford University, Stanford, CA (2014).Google Scholar
- Christian Hansen and Damian Kozbur. 2014. Instrumental variables estimation with many weak instruments using regularized JIVE. Journal of Econometrics Vol. 182, 2 (2014), 290--308.Google ScholarCross Ref
- Jason Hartford, Greg Lewis, Kevin Leyton-Brown, and Matt Taddy. 2016. Counterfactual Prediction with Deep Instrumental Variables Networks. arXiv preprint arXiv:1612.09596 (2016).Google Scholar
- Lars G Hemkens, Despina G Contopoulos-Ioannidis, and John PA Ioannidis. 2016. Agreement of treatment effects for mortality from routinely collected data and subsequent randomized trials: Meta-epidemiological survey. British Medical Journal Vol. 352 (2016).Google Scholar
- Kosuke Imai, Dustin Tingley, and Teppei Yamamoto. 2013. Experimental designs for identifying causal mechanisms. Journal of the Royal Statistical Society: Series A (Statistics in Society), Vol. 176, 1 (2013), 5--51.Google ScholarCross Ref
- Guido Imbens, Joshua Angrist, and Alan Krueger. 1999. Jackknife Instrumental Variables Estimation. Journal of Applied Econometrics Vol. 14, 1 (1999).Google Scholar
- Jongbin Jung, Connor Concannon, Ravi Shroff, Sharad Goel, and Daniel G Goldstein. 2017. Simple rules for complex decisions. arXiv preprint arXiv:1702.04690 (2017).Google Scholar
- Ron Kohavi, Alex Deng, Brian Frasca, Toby Walker, Ya Xu, and Nils Pohlmann. 2013. Online controlled experiments at large scale. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 1168--1176. Google ScholarDigital Library
- Robert J LaLonde. 1986. Evaluating the econometric evaluations of training programs with experimental data. The American Economic Review (1986), 604--620.Google Scholar
- Finnian Lattimore, Tor Lattimore, and Mark D Reid. 2016. Causal Bandits: Learning Good Interventions via Causal Inference Advances in Neural Information Processing Systems. 1181--1189. Google ScholarDigital Library
- Lihong Li, Wei Chu, John Langford, and Robert E Schapire. 2010. A contextual-bandit approach to personalized news article recommendation Proceedings of the 19th international conference on World wide web. ACM, 661--670. Google ScholarDigital Library
- Michelle N Meyer. 2015. Two cheers for corporate experimentation: The A/B illusion and the virtues of data-driven innovation. J. on Telecomm. & High Tech. L. Vol. 13 (2015), 273.Google Scholar
- Art B. Owen. 2016. Monte Carlo Theory, Methods and Examples. http://statweb.stanford.edu/ owen/mc/Google Scholar
- Judea Pearl. 2009. Causality. Cambridge University Press.Google Scholar
- Alexander Peysakhovich and Jeffrey Naecker. 2017. Using methods from machine learning to evaluate behavioral models of choice under risk and ambiguity. Journal of Economic Behavior & Organization Vol. 133 (2017), 373--384.Google ScholarCross Ref
- Olav Reiersöl. 1945. Confluence analysis by means of instrumental sets of variables. Ph.D. Dissertation. bibinfoschoolStockholm College.Google Scholar
- Uri Shalit, Fredrik Johansson, and David Sontag. 2016. Bounding and Minimizing Counterfactual Error. arXiv preprint arXiv:1606.03976 (2016).Google Scholar
- Douglas Staiger and James H Stock. 1997. Instrumental Variables Regression with Weak Instruments. Econometrica (1997), 557--586.Google Scholar
- James H Stock, Jonathan H Wright, and Motohiro Yogo. 2012. A survey of weak instruments and weak identification in generalized method of moments. Journal of Business & Economic Statistics (2012).Google Scholar
- James H Stock and Motohiro Yogo. 2005. Testing for weak instruments in linear IV regression. Identification and Inference for Econometric Models: Essays in Honor of Thomas Rothenberg. Cambridge University Press, 80--108.Google Scholar
- Richard S Sutton and Andrew G Barto. 1998. Reinforcement learning: An introduction. Vol. Vol. 1. MIT press Cambridge. Google ScholarDigital Library
- Hal Varian. 2016. Intelligent Technology. Finance and Development Vol. 53, 3 (2016).Google Scholar
- Jeffrey M Wooldridge. 2010. Econometric Analysis of Cross Section and Panel Data. MIT Press.Google Scholar
- Philip Green Wright. 1928. The Tariff on Animal and Vegetable Oils. The Macmillan Co.Google Scholar
- Ya Xu, Nanyu Chen, Addrian Fernandez, Omar Sinno, and Anmol Bhasin. 2015. From infrastructure to culture: A/B testing challenges in large scale social networks Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 2227--2236. Google ScholarDigital Library
Index Terms
- Learning Causal Effects From Many Randomized Experiments Using Regularized Instrumental Variables
Recommendations
Conditions Sufficient to Infer Causal Relationships Using Instrumental Variables and Observational Data
Econometritions frequently believe that standard instrumental variables (IV) methods can prove causal relationships. We review the relevant formal causal inference literature, and we demonstrate that this belief is not justified. Couching the problem in ...
Learning instrumental variables with structural and non-gaussianity assumptions
Learning a causal effect from observational data requires strong assumptions. One possible method is to use instrumental variables, which are typically justified by background knowledge. It is possible, under further assumptions, to discover whether a ...
Semi-instrumental variables: a test for instrument admissibility
UAI'01: Proceedings of the Seventeenth conference on Uncertainty in artificial intelligenceIn a causal graphical model, an instrument for a variable X and its effect Y is a random variable that is a cause of X and independent of all the causes of Y except X (Pearl 1995). For continuous variables, instrumental variables can be used to estimate ...
Comments