Nikunj Saunshi
I am a researcher at Google Research NYC. My interests lie in using a mix of theory and empirics to demystify the incredible success of deep learning and AI, and to design efficient and reliable learning algorithms. My research has spanned topics like language models, self-supervised learning, representation learning, meta-learning, natural language processing, interpretability of deep learning models.
I received my PhD in Computer Science from Princeton University, where I was advised by Sanjeev Arora. Before joining Princeton, I worked in the R&D center of Samsung Electronics in Suwon, South Korea. I completed my B.Tech. with Honors in Computer Science and Engineering and Minor in Mathematics from the Indian Institute of Technology Bombay.
Updates
[May 2023] Reasoning in Large Language Models Through Symbolic Math Word Problems accepted at Findings of ACL 2023 and Natural Language Reasoning and Structured Explanations (NLRSE) workshop
[Apr 2023] Task-Specific Skill Localization in Fine-tuned Language Models accepted at ICML 2023!
[Apr 2023] Understanding Influence Functions and Datamodels via Harmonic Analysis presented at ICLR 2023
[Apr 2023] Selected as Notable Reviewer for ICLR 2023
[Jan 2023] Started as a Research Scientist at Google
[Jan 2023] Presented a talk on Towards Understanding Self-Supervised Learning at CMU
[July 2022] Defended my PhD thesis!
Publications by year
(#) denotes alphabetical order
Efficient Stagewise Pretraining via Progressive Subnetworks
Abhishek Panigrahi*, Nikunj Saunshi*, Kaifeng Lyu, Sobhan Miryoosefi, Sashank Reddi, Satyen Kale, Sanjiv Kumar
Arxiv
Task-Specific Skill Localization in Fine-tuned Language Models
Abhishek Panigrahi*, Nikunj Saunshi*, Haoyu Zhang, Sanjeev Arora
To appear in ICML 2023
Reasoning in Large Language Models Through Symbolic Math Word Problems
Vedant Gaur, Nikunj Saunshi
Findings of ACL 2023
Understanding Influence Functions and Datamodels via Harmonic Analysis
Nikunj Saunshi, Arushi Gupta, Mark Braverman, Sanjeev Arora
ICLR 2023
New Definitions and Evaluations for Saliency Methods: Staying Intrinsic, Complete and Sound
Arushi Gupta*, Nikunj Saunshi*, Dingli Yu*, Kaifeng Lyu, Sanjeev Arora
NeurIPS 2022 (Oral)
Understanding Contrastive Learning Requires Incorporating Inductive Biases
Nikunj Saunshi, Jordan Ash, Surbhi Goel, Dipendra Misra, Cyril Zhang, Sanjeev Arora, Sham Kakade, Akshay Krishnamurthy
ICML 2022
On Predicting Generalization using GANs
Yi Zhang, Arushi Gupta, Nikunj Saunshi, Sanjeev Arora
ICLR 2022 (Oral)
Predicting What You Already Know Helps: Provable Self-Supervised Learning
(#) Jason D. Lee, Qi Lei, Nikunj Saunshi, Jiacheng Zhuo
NeurIPS 2021
Nikunj Saunshi, Arushi Gupta, Wei Hu
ICML 2021
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Nikunj Saunshi, Sadhika Malladi, Sanjeev Arora
ICLR 2021
[Talk]
A Sample Complexity Separation between Non-Convex and Convex Meta-Learning
Nikunj Saunshi, Yi Zhang, Mikhail Khodak, Sanjeev Arora
ICML 2020
[Talk]
Provable Representation Learning for Imitation Learning via Bi-level Optimization
(#) Sanjeev Arora, Simon S. Du, Sham Kakade, Yuping Luo, Nikunj Saunshi
ICML 2020
A Theoretical Analysis of Contrastive Unsupervised Representation Learning
(#) Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, Nikunj Saunshi
ICML 2019
A La Carte Embedding: Cheap but Effective Induction of Semantic Feature Vectors
Mikhail Khodak*, Nikunj Saunshi*, Yingyu Liang, Tengyu Ma, Brandon Stewart and Sanjeev Arora
ACL 2018
A Compressed Sensing View of Unsupervised Text Embeddings, Bag-of-n-Grams, and LSTMs
(#) Sanjeev Arora, Mikhail Khodak, Nikunj Saunshi, Kiran Vodrahalli
ICLR 2018
[Blog]
Publications by topics
Language models and Self-supervised learning
(Language models, contrastive learning)
Efficient Stagewise Pretraining via Progressive Subnetworks
Abhishek Panigrahi*, Nikunj Saunshi*, Kaifeng Lyu, Sobhan Miryoosefi, Sashank Reddi, Satyen Kale, Sanjiv Kumar
Arxiv
Task-Specific Skill Localization in Fine-tuned Language Models
Abhishek Panigrahi*, Nikunj Saunshi*, Haoyu Zhang, Sanjeev Arora
To appear in ICML 2023
Reasoning in Large Language Models Through Symbolic Math Word Problems
Vedant Gaur, Nikunj Saunshi
Findings of ACL 2023
Understanding Contrastive Learning Requires Incorporating Inductive Biases
Nikunj Saunshi, Jordan Ash, Surbhi Goel, Dipendra Misra, Cyril Zhang, Sanjeev Arora, Sham Kakade, Akshay Krishnamurthy
ICML 2022
Predicting What You Already Know Helps: Provable Self-Supervised Learning
(#) Jason D. Lee, Qi Lei, Nikunj Saunshi, Jiacheng Zhuo
NeurIPS 2021
A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Nikunj Saunshi, Sadhika Malladi, Sanjeev Arora
ICLR 2021
[Talk]
A Theoretical Analysis of Contrastive Unsupervised Representation Learning
(#) Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, Nikunj Saunshi
ICML 2019
Interpretability
Understanding Influence Functions and Datamodels via Harmonic Analysis
Nikunj Saunshi, Arushi Gupta, Mark Braverman, Sanjeev Arora
ICLR 2023
New Definitions and Evaluations for Saliency Methods: Staying Intrinsic, Complete and Sound
Arushi Gupta*, Nikunj Saunshi*, Dingli Yu*, Kaifeng Lyu, Sanjeev Arora
NeurIPS 2022 (Oral)
Meta learning
Nikunj Saunshi, Arushi Gupta, Wei Hu
ICML 2021
A Sample Complexity Separation between Non-Convex and Convex Meta-Learning
Nikunj Saunshi, Yi Zhang, Mikhail Khodak, Sanjeev Arora
ICML 2020
[Talk]
Provable Representation Learning for Imitation Learning via Bi-level Optimization
(#) Sanjeev Arora, Simon S. Du, Sham Kakade, Yuping Luo, Nikunj Saunshi
ICML 2020
Representation learning & NLP
A La Carte Embedding: Cheap but Effective Induction of Semantic Feature Vectors
Mikhail Khodak*, Nikunj Saunshi*, Yingyu Liang, Tengyu Ma, Brandon Stewart and Sanjeev Arora
ACL 2018
A Compressed Sensing View of Unsupervised Text Embeddings, Bag-of-n-Grams, and LSTMs
(#) Sanjeev Arora, Mikhail Khodak, Nikunj Saunshi, Kiran Vodrahalli
ICLR 2018
[Blog]
Other topics
On Predicting Generalization using GANs
Yi Zhang, Arushi Gupta, Nikunj Saunshi, Sanjeev Arora
ICLR 2022 (Oral)