Ali Safaya

I am an AI Research Scientist at HyperbeeAI. I received my PhD in Computer Science from Koç University in 2026, where I was advised by Deniz Yuret and was an AI Research Fellow at the KUIS AI Center in Istanbul. My thesis, Vector Retrieval-Augmented Language Models: What Retrieval Can and Cannot Replace, measured where retrieval can stand in for a language model's parameters: it can for knowledge, but not for reasoning. My research focuses on memory augmentation for language models.

Email  /  CV (EN / AR)  /  Semantic Scholar  /  Google Scholar  /  ORCID  /  GitHub  /  Hugging Face  /  X

I am interested in natural language processing, language modeling (both large and small models), and deep learning in general. My PhD research studied vector retrieval as external memory for language models, both episodic memory of a long document the model is reading and semantic memory of general world knowledge. Both rest on Neurocache, which retrieves hidden states instead of text: it runs k-nearest-neighbor search over cached states at a middle layer of a frozen pre-trained model and passes what it finds to the upper layers through a separate cache-attention module. Against text retrieval, neither side wins everywhere: the cache loses open-web question answering, but wins lexical and structured lookup, and wins multi-hop questions for far less computation. A small model with this memory beats a larger model trained the same way on knowledge questions, and editing the store changes the model's answers with no weight update. Its main limitation is that it cannot tell when what it retrieved is wrong, or when the answer is absent.

From 2021 to 2024, I was also an active member of the Turkish Data Depository project, an open-source repository for data and tools for the Turkish language, where I managed TDD's dataset hub and oversaw Mukayese, a collection of Turkish language benchmarks.

Neurocache extends the effective context of large language models with an external vector cache of compressed past states, retrieved through efficient k-nearest-neighbor lookup and fused back into the attention layers.

Kanarya is a 2-billion-parameter Turkish language model based on the GPT-J architecture, pretrained on large-scale filtered Turkish web text and released openly for Turkish text generation and NLP research.

Mukayese is a collection of NLP benchmarks for the Turkish language, consisting of seven leaderboards for different NLP tasks. For each benchmark, we work with one or more datasets and present two or more baselines.

HuBERT-TR: Reviving Turkish Automatic Speech Recognition with Self-supervised Speech Representation Learning
Ali Safaya, Engin Erzin · arXiv, 2022

Beyond the Imitation Game: Quantifying and Extrapolating the Capabilities of Language Models (BIG-bench)
Aarohi Srivastava et al. (incl. Ali Safaya) · TMLR, 2023

Event Coreference Resolution for Contentious Politics Events
Ali Hurriyetoglu et al. (incl. Ali Safaya) · arXiv, 2022

KUISAIL at SemEval-2020 Task 12: BERT-CNN for Offensive Speech Identification in Social Media
Ali Safaya, Moutasem Abdullatif, Deniz Yuret · SemEval-2020

Automated Extraction of Socio-political Events from News (AESPEN): Workshop and Shared Task Report
Ali Hurriyetoglu et al. (incl. Ali Safaya) · AESPEN, 2020

COVCOR20 at WNUT-2020 Task 2: An Attempt to Combine Deep Learning and Expert Rules
Ali Hurriyetoglu et al. (incl. Ali Safaya) · W-NUT, 2020

Event Sentence Detection Task using Attention Model
Ali Safaya · CLEF, 2019

Understanding the Impact of Token Redundancy in Language Models
How subword tokenizers (BPE, SentencePiece) end up learning duplicate tokens for the same word depending on a leading space, why that wastes capacity, and how models still learn to treat the duplicates alike — shown through embedding cosine-similarity heatmaps.

BERT-CNN for Detecting Offensive Speech
A BERT + CNN architecture for identifying offensive language in social media, from our SemEval-2020 Task 12 (OffensEval) submission.

Arabic-ALBERT: Pre-training Arabic Language Models
Open-source ALBERT language models pre-trained for Arabic, released with code and weights for the community.

This website is based on Jon Barron website's source code.