Legal Clause Classification
Automatically labelling contract clauses, and choosing a metric that does not flatter the model.
- Timeline
- Sep 2025 – Mar 2026 · Complete
- Role
- Led a team of three
- Modality
- Text
- Evaluated on
- Macro F1 as the gating metric
- Macro F1, feedforward
- 0.9296
- Macro F1, LSTM
- 0.9205
- Team size (led)
- 3
- Benchmark
- CUAD
The problem
Contract review is slow and inconsistent because a human has to read every clause and decide what kind of clause it is. The CUAD benchmark exists precisely because this is hard: clause types are wildly imbalanced, and the rare ones are often the ones that matter legally.
What I built
I led a three-person team building an end-to-end NLP pipeline that classifies contract clauses (indemnification, termination, confidentiality, and more) from CUAD. Real contracts are messy, so a lot of the work was preprocessing: sentence-boundary detection across legal punctuation, clause segmentation via section headers, handling citations and cross-references, and a strategy for documents too long to process in one pass. Built with NLTK for tokenization and lemmatization, scikit-learn for TF-IDF, and TensorFlow/Keras for the models. We compared a TF-IDF + feedforward network against an LSTM over token sequences.
How I knew it worked
The TF-IDF feedforward model won at 93.19% accuracy and 0.9296 macro F1, against 92.17% and 0.9205 for the LSTM. The metric choice was the real decision. Accuracy on an imbalanced clause distribution mostly measures performance on the common clause types, which hides degradation on the rare ones. Macro F1 weights every clause type equally, so it exposes exactly the long-tail failure that accuracy conceals. I made macro F1 the gating metric and compared the two architectures under it rather than under the more flattering number. The margin is small, which is the point: a metric that can separate two close models is doing more work than one that flatters both.
