Publications

2026

1. 

   Heterogeneity in Formal Linguistic Competence of Language Models: Is Data the Real Bottleneck?

   H S V N S Kowndinya Renduchintala^†, and Sumit Bhatia

   arXiv preprint arXiv:2604.17930, 2026

   Abs Bib PDF

   

   Large Language Models (LLMs) exhibit a puzzling disparity in their formal linguistic competence: while they learn some linguistic phenomena with near-perfect mastery, they often perform below chance on others, even after training on trillions of tokens. In this work, we investigate whether these failures stem from inherent architectural limitations or simply the scarcity of these specific grammatical constructions in web-scale corpora. We pre-train simple GPT-2 Small (124M) models on a 100M-token random sample of the FineWeb corpus and intervene by injecting a minimal amount (1%) of synthetic data targeting specific linguistic phenomena. We find that this targeted intervention substantially improves model performance in 8 out of the 9 worst-performing BLiMP paradigms - notably the accuracy on a specific paradigm, only_npi_scope, surges from 20.9% to 69.4%. Furthermore, we observe that these interventions generally preserve or slightly improve aggregate performance. However, while we also identify a resistant phenomenon, principle_A_c_command, whose performance remains below chance even after our data augmentation, our findings do serve as an optimistic existence proof that even small language models can substantially improve on those linguistic phenomena on which models typically perform poorly, provided the pre-training data contains sufficient exposure to them. This suggests that efforts towards human-scale language modeling may benefit greatly by focusing on data composition. The code to reproduce our results is open-sourced at https://github.com/kowndinya-renduchintala/heterogeneity-in-formal-linguistic-competence.

   @article{renduchintala2026heterogeneity,
     title = {Heterogeneity in Formal Linguistic Competence of Language Models: Is Data the Real Bottleneck?},
     author = {Renduchintala, H S V N S Kowndinya and Bhatia, Sumit},
     journal = {arXiv preprint arXiv:2604.17930},
     year = {2026},
     url = {https://arxiv.org/abs/2604.17930},
     google_scholar_id = {3fE2CSJIrl8C}
   }

2025

1. 

   On the Effect of Instruction Tuning Loss on Generalization

   Anwoy Chatterjee^†, H S V N S Kowndinya Renduchintala^†, Sumit Bhatia, and 1 more author

   Transactions of the Association for Computational Linguistics, Oct 2025

   Abs Bib PDF Blog

   

   Instruction tuning has emerged as a pivotal post-training paradigm that enables pre-trained language models to better follow user instructions. Despite its significance, little attention has been given to optimizing the loss function used. A fundamental, yet often overlooked, question is whether the conventional auto-regressive objective—where loss is computed only on response tokens, excluding prompt tokens—is truly optimal for instruction tuning. In this work, we systematically investigate the impact of differentially weighting prompt and response tokens in instruction tuning loss, and propose Weighted Instruction Tuning (WIT) as a better alternative to conventional instruction tuning. Through extensive experiments on five language models of different families and scale, three finetuning datasets of different sizes, and five diverse evaluation benchmarks, we show that the standard instruction tuning loss often yields suboptimal performance and limited robustness to input prompt variations. We find that a low-to-moderate weight for prompt tokens coupled with a moderate-to-high weight for response tokens yields the best-performing models across settings and also serves as a better starting point for the subsequent preference alignment training. These findings highlight the need to reconsider instruction-tuning loss and offer actionable insights for developing more robust and generalizable models. Our code is open-sourced at https://github.com/kowndinya-renduchintala/WIT.

   @article{10.1162/TACL.a.42,
     author = {Chatterjee, Anwoy and Renduchintala, H S V N S Kowndinya and Bhatia, Sumit and Chakraborty, Tanmoy},
     title = {On the Effect of Instruction Tuning Loss on
                         Generalization},
     journal = {Transactions of the Association for Computational Linguistics},
     volume = {13},
     pages = {1360-1380},
     year = {2025},
     month = oct,
     issn = {2307-387X},
     doi = {10.1162/TACL.a.42},
     url = {https://doi.org/10.1162/TACL.a.42},
     eprint = {https://direct.mit.edu/tacl/article-pdf/doi/10.1162/TACL.a.42/2561636/tacl.a.42.pdf},
     google_scholar_id = {UebtZRa9Y70C},
   }

2024

1. 

   POSIX: A Prompt Sensitivity Index For Large Language Models

   Anwoy Chatterjee^†, H S V N S Kowndinya Renduchintala^†, Sumit Bhatia, and 1 more author

   In Findings of the Association for Computational Linguistics: EMNLP 2024, Nov 2024

   Abs Bib PDF Poster

   

   Despite their remarkable capabilities, Large Language Models (LLMs) are found to be surprisingly sensitive to minor variations in prompts, often generating significantly divergent outputs in response to minor variations in the prompts, such as spelling errors, alteration of wording or the prompt template. However, while assessing the quality of an LLM, the focus often tends to be solely on its performance on downstream tasks, while very little to no attention is paid to prompt sensitivity. To fill this gap, we propose POSIX – a novel PrOmpt Sensitivity IndeX as a reliable measure of prompt sensitivity, thereby offering a more comprehensive evaluation of LLM performance. The key idea behind POSIX is to capture the relative change in loglikelihood of a given response upon replacing the corresponding prompt with a different intent-preserving prompt. We provide thorough empirical evidence demonstrating the efficacy of POSIX in capturing prompt sensitivity and subsequently use it to measure and thereby compare prompt sensitivity of various open source LLMs. We find that merely increasing the parameter count or instruction tuning does not necessarily reduce prompt sensitivity whereas adding some few-shot exemplars, even just one, almost always leads to significant decrease in prompt sensitivity. We also find that alterations to prompt template lead to the highest sensitivity in the case of MCQ type tasks, whereas paraphrasing results in the highest sensitivity in open-ended generation tasks. The code for reproducing our results is open-sourced at https://github.com/kowndinya-renduchintala/POSIX.

   @inproceedings{chatterjee-etal-2024-posix,
     title = {{POSIX}: A Prompt Sensitivity Index For Large Language Models},
     author = {Chatterjee, Anwoy and Renduchintala, H S V N S Kowndinya and Bhatia, Sumit and Chakraborty, Tanmoy},
     editor = {Al-Onaizan, Yaser and Bansal, Mohit and Chen, Yun-Nung},
     booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2024},
     month = nov,
     year = {2024},
     address = {Miami, Florida, USA},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2024.findings-emnlp.852},
     pages = {14550--14565},
     google_scholar_id = {Se3iqnhoufwC},
   }

2. 

   SMART: Submodular Data Mixture Strategy for Instruction Tuning

   H S V N S Kowndinya Renduchintala, Sumit Bhatia, and Ganesh Ramakrishnan

   In Findings of the Association for Computational Linguistics ACL 2024, Aug 2024

   Abs Bib PDF Video Poster

   

   Instruction Tuning involves finetuning a language model on a collection of instruction-formatted datasets in order to enhance the generalizability of the model to unseen tasks. Studies have shown the importance of balancing different task proportions during finetuning, but finding the right balance remains challenging. Unfortunately, there’s currently no systematic method beyond manual tuning or relying on practitioners’ intuition. In this paper, we introduce SMART (Submodular data Mixture strAtegy for instRuction Tuning) — a novel data mixture strategy which makes use of a submodular function to assign importance scores to tasks which are then used to determine the mixture weights. Given a fine-tuning budget, SMART redistributes the budget among tasks and selects non-redundant samples from each task. Experimental results demonstrate that SMART significantly outperforms traditional methods such as examples proportional mixing and equal mixing. Furthermore, SMART facilitates the creation of data mixtures based on a few representative subsets of tasks alone and through task pruning analysis, we reveal that in a limited budget setting, allocating budget among a subset of representative tasks yields superior performance compared to distributing the budget among all tasks. The code for reproducing our results is open-sourced at https://github.com/kowndinya-renduchintala/SMART.

   @inproceedings{renduchintala-etal-2024-smart,
     title = {{SMART}: Submodular Data Mixture Strategy for Instruction Tuning},
     author = {Renduchintala, H S V N S Kowndinya and Bhatia, Sumit and Ramakrishnan, Ganesh},
     editor = {Ku, Lun-Wei and Martins, Andre and Srikumar, Vivek},
     booktitle = {Findings of the Association for Computational Linguistics ACL 2024},
     month = aug,
     year = {2024},
     address = {Bangkok, Thailand and virtual meeting},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2024.findings-acl.766},
     doi = {10.18653/v1/2024.findings-acl.766},
     pages = {12916--12934},
     google_scholar_id = {qjMakFHDy7sC},
   }

2023

1. 

   INGENIOUS: Using Informative Data Subsets for Efficient Pre-Training of Language Models

   H S V N S Kowndinya Renduchintala, Krishnateja Killamsetty, Sumit Bhatia, and 4 more authors

   In Findings of the Association for Computational Linguistics: EMNLP 2023, Dec 2023

   Abs Bib PDF Video Poster

   

   A salient characteristic of pre-trained language models (PTLMs) is a remarkable improvement in their generalization capability and emergence of new capabilities with increasing model capacity and pre-training dataset size. Consequently, we are witnessing the development of enormous models pushing the state-of-the-art. It is, however, imperative to realize that this inevitably leads to prohibitively long training times, extortionate computing costs, and a detrimental environmental impact. Significant efforts are underway to make PTLM training more efficient through innovations in model architectures, training pipelines, and loss function design, with scant attention being paid to optimizing the utility of training data. The key question that we ask is whether it is possible to train PTLMs by employing only highly informative subsets of the training data while maintaining downstream performance? Building upon the recent progress in informative data subset selection, we show how we can employ submodular optimization to select highly representative subsets of the training corpora and demonstrate that the proposed framework can be applied to efficiently train multiple PTLMs (BERT, BioBERT, GPT-2) using only a fraction of data. Further, we perform a rigorous empirical evaluation to show that the resulting models achieve up to ~99% of the performance of the fully-trained models. We made our framework publicly available at https://github.com/Efficient-AI/ingenious.

   @inproceedings{renduchintala-etal-2023-ingenious,
     title = {{INGENIOUS}: Using Informative Data Subsets for Efficient Pre-Training of Language Models},
     author = {Renduchintala, H S V N S Kowndinya and Killamsetty, Krishnateja and Bhatia, Sumit and Aggarwal, Milan and Ramakrishnan, Ganesh and Iyer, Rishabh and Krishnamurthy, Balaji},
     editor = {Bouamor, Houda and Pino, Juan and Bali, Kalika},
     booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2023},
     month = dec,
     year = {2023},
     address = {Singapore},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2023.findings-emnlp.445},
     doi = {10.18653/v1/2023.findings-emnlp.445},
     pages = {6690--6705},
     google_scholar_id = {ufrVoPGSRksC},
   }