06 — NLP¶
Natural language processing: preprocessing, tokenization, classical NLP, and modern transformer-based NLP.
Scope¶
Cheat sheets for text processing, classical NLP tools (NLTK, spaCy), and neural NLP (Hugging Face, sentence-transformers). LLM-specific topics (prompting, fine-tuning APIs) belong to 08-LLM.
Cheat Sheets¶
| Sheet | Covers |
|---|---|
| nlp-preprocessing | Full preprocessing pipeline, when to stem/lemma, messy text, transformer vs classical |
| nltk.md | Tokenization, stopwords, stemming, lemmatization, POS, VADER sentiment |
| spacy.md | Fast NLP: POS, NER, dependency parsing, Matcher, transformer integration |
| huggingface-transformers | Pipelines, AutoModel, BERT/GPT/T5, fine-tuning with Trainer |
| huggingface-datasets | load_dataset, map, streaming, tokenization, upload |
| tokenizers | BPE, WordPiece, Unigram, training from scratch, alignment |
| sentence-transformers | Semantic search, embeddings, cross-encoder re-ranking |
Contribute¶
- Template: CHEATSHEET_TEMPLATE.md
- Guide: CONTRIBUTING.md
- Planned: text classification deep-dive, multilingual NLP, information extraction