🚀 NeuroText: Transformer-Based Next Word Prediction

NeuroText is a custom Transformer-based Language Model designed for next-word prediction. The project implements a decoder-style Transformer architecture from scratch using TensorFlow/Keras, incorporating Multi-Head Self-Attention, Positional Encoding, Layer Normalization, and Feed-Forward Networks.

The model learns contextual relationships between words and predicts the most probable next word given a sequence of previous words.

---

✨ Features

• Custom Transformer architecture built from scratch
• Sinusoidal Positional Encoding
• Multi-Head Self-Attention with Causal Masking
• Pre-Normalization Transformer Blocks
• GELU Activation Function
• Residual Connections
• Dropout Regularization
• Early Stopping & Best Model Checkpointing
• Top-K Accuracy Evaluation
• Perplexity-Based Language Model Assessment

---

🏗️ Architecture Overview

Input Layer

• Tokenized text sequences
• Fixed context length

Embedding Layer

Converts token IDs into dense vector representations.

Positional Encoding

Adds positional information using sinusoidal encoding:

Transformer Blocks

Each Transformer block contains:

1. Multi-Head Self-Attention
2. Layer Normalization (Pre-Norm)
3. Residual Connection
4. Feed Forward Network
5. Dropout

Output Layer

• Global Average Pooling
• Layer Normalization
• Dense Softmax Layer

Produces probability distribution over the entire vocabulary.

---

⚙️ Model Configuration

Parameter	Value
Embedding Dimension	128
Transformer Blocks	4
Attention Heads	4
Feed Forward Dimension	512
Dropout	0.1
Activation	GELU
Output Layer	Softmax
Optimizer	AdamW
Loss Function	Sparse Categorical Crossentropy

---

🧠 Custom Components

Positional Encoding

Custom sinusoidal positional encoding layer implemented from scratch.

Transformer Block

Custom decoder-style Transformer block featuring:

• Causal Self-Attention
• Layer Normalization
• Residual Connections
• Feed Forward Network
• Dropout Regularization

---

📊 Model Performance

Top-K Accuracy

Metric	Score
Top-1 Accuracy	28.19%
Top-3 Accuracy	44.74%
Top-5 Accuracy	51.41%
Top-10 Accuracy	59.83%

Language Modeling Metrics

Metric	Value
Perplexity	55.94

Performance Summary

• Correct prediction appears as the highest probability word 28.19% of the time.
• Correct prediction appears within the top 3 predictions 44.74% of the time.
• Correct prediction appears within the top 5 predictions 51.41% of the time.
• Correct prediction appears within the top 10 predictions 59.83% of the time.
• Perplexity of 55.94 indicates strong contextual learning for a custom Transformer trained from scratch.

---

📈 Evaluation Metrics

Top-K Accuracy

Measures whether the correct next word appears among the top K predicted words.

Perplexity

Perplexity is a standard metric for language models:

[ Perplexity = e^{Loss} ]

Lower perplexity indicates better language modeling performance.

---

🔍 Example Prediction

Input

Artificial intelligence is transforming the

Model Prediction

world
future
industry
way
technology

---

🛠️ Tech Stack

• Python
• TensorFlow / Keras
• NumPy
• Matplotlib
• Google Colab

---

🚀 Future Improvements

• Larger vocabulary size
• Deeper Transformer stacks
• Byte Pair Encoding (BPE)
• Subword tokenization
• Beam Search Decoding
• Attention Visualization
• GPT-style autoregressive generation
• Fine-tuning on domain-specific corpora

---

🎯 Key Achievement

NeuroText demonstrates that a custom-built Transformer architecture can effectively learn contextual language patterns and achieve competitive next-word prediction performance while being significantly smaller and more lightweight than large-scale language models.

---

👩‍💻 Author

Shruti Patel

Transformer-Based Language Modeling Project
Built with TensorFlow and Transformer Architecture from Scratch.