NeurIPS 2019的一些观感

前几天有提到，十二月份的时候去NeurIPS 2019晃了一圈。除了开篇那个演讲之外，那周我还去围观了不少其他的东西。NeurIPS开到如今，万人大会，熙熙攘攘地其实挺难甄别信息的。第一次去这种顶级的计算机会议，经验不足，只能按图索骥般地一点点拾遗。

好在，我目标相对明确，并不是一味的去凑热闹的。那些火到爆的GAN之类的，我就只能远远地围观一眼，然后不明觉厉，去找自己相对更能看懂的东西去了。于是，我就很无聊的，去看了两个主题：因果推断相关的，以及隐私相关的。

因果推断这块儿，能到NeurIPS自然是被选择过的，不会是太纯理论的这种。跟机器学习相关的自然是要跳出简单的线性回归了，否则大家写什么呢？其次呢，就是跳出随机试验的框架，否则哪里用得到那么多高深的预测模型呢？七七八八看了不少poster论文，大部分都是各种花样繁杂的算法。努力地去理解他们的做法，然而却哀叹一声，浮沙筑高台，又有多少可以大浪淘沙始见金。（插曲：后面那个causal inference workshop，直接就是Susan-fest...哎，她也算是扛起来一面大旗了。）

隐私相关的，其实是加密+分布式的结合，基本要实现的是在客户端进行计算而不是需要把原始的隐私数据传递到服务器端。于是乎，我们看到了各种基于分布式算法的演化，加一些随机的因素在里面，就得到了一些隐私友好的算法。也挺好玩的，有助于想清楚一些分布式算法的设计。

笔记本身是用英文整理的，直接在这里贴一份好了。

Causal ML

Theme 1:  causal inference (regression) in a non-linear or ML setting. 

• Microsoft: Machine learning estimation of heterogeneous treatment effects with instruments
  • Reducing the heterogeneous treatment effect problem to the minimization of an appropriate loss function that depends on a set of auxiliary models (each corresponding to a separate prediction task). The reduction enables the use of all recent algorithmic advances (e.g. neural nets, forests)
• MIT economics: Kernel instrumental variable regression
  • Use kernel ridge regression in traditional 2SLS 
• Columbia Univ: Adapting Neural Networks for the Estimation of Treatment Effects
  • Neural network models to improve estimation of propensity score and expected outcome (potential outcome)

• AWS: Causal regularization
  • regularization is even recommended in the infinite sample regime because it helps making predictive models more causal, when the effect of multivariate confounders can be so similar to finite sample effects that the same techniques that avoid overfitting also temper the impact of confounding.
• Google: Variance reduction in bipartite experiments through correlation clustering
  • the units of randomization and the units of analysis are bipartite, then use clustering to maximize the statistical power of a bipartite experiment.
• Columbia Univ: Thompson sampling for multinomial logit contextual bandits 
  • Feedback is given by a multinomial logit choice model in a bandit game 
• Uber: Variational Bayesian Optimal Experimental Design 
  • Variational estimators of expected information gain 
• IBM: Characterization and learning of causal graphs with latent variable from soft interventions

Theme 2: non-experimental based causal inference

• MIT: Synthetic control tutorial (video)
  • An intro tutorial on synthetic control to ML people 
• King’s College London: Debiased Bayesian inference for average treatment effects 
  • Bayesian (and other) methods can be badly biased  in complex models or with high-dimensional features, hence introduce a prior based on the propensity score that corrects for the first-order posterior bias
• Cornell Univ: Deep generalized method of moments for instrumental variable analysis
  •  IV with such high-dimensional variables / complex relationships
• Purdue Univ: Efficient Identification in Linear Structural Causal Models with Instrumental Cutsets
  • Find instrumental Sets

Theme 3: Computational progress and auction-related , etc. (also link to part 1 PPML)

• University of Haifa, Israel: Fast and accurate least-mean-squares solvers, 
  • boost the performance of existing LMS solvers, such as those in scikit-learn library, up to x100.
• MIT EECS: On Robustness of principal component regression 
  • PCR’s property: robust to noisy, sparse, and possibly mixed valued covariates. 
• UC Berkeley: A meta-analysis of overfitting in machine learning
  • Assess how much participants adapted to the holdout set over the course of a competition and verified the robustness of the holdout method across different data domains, loss functions, model classes, and human analysts.
• Google: Prior-free dynamic auctions with low regret buyers.
  • In the prior-free setting where the buyer’s value each round is chosen adversarially (possibly adaptively), it is possible to extract an approximation of the full economic surplus.

Causality workshop led by Susan Athey: video 

Privacy-preserving (parallel computing + local noise/encryption ) 

• Google: Differentially Private Covariance Estimation
  •  Privately estimate the unnormalized covariance matrix
• Differentially Private Bayesian Linear Regression

Others

• How to learn by Celeste Kidd, UC Berkeley
• Veridical data science by Bin Yu, UC Berkeley. 
  • Predictability
  • Computability 
  • Stability 
• Fairness and measurement of AI 
  • Measurement and fairness,   https://arxiv.org/abs/1912.05511 
  • Fairness-related harms can occur, including problematic training data (e.g., deriving credit scores from data about racist lending practices), questionable modeling choices (e.g., modeling teacher quality as only a function of students’ test scores), and lack of due process (e.g., automating consequential decisions about people’s lives without due process). -> operationalized via a measurement model.
• Power analysis of knockoff filters for correlated designs