Applied NLP: Embeddings and Vector Databases

Explain at a high level how the Transformer architecture works.

Use a pre-trained model from Hugging Face to generate text embeddings.

Explain the concept of a vector database and its role in semantic search.

Set up a local vector database (e.g., ChromaDB) to store and index document embeddings.

Ingest a corpus of text documents, chunk them, and store their embeddings in the vector database.

Perform semantic search by converting a query to an embedding and retrieving the most similar vectors.

Explain the architecture of a Retrieval-Augmented Generation (RAG) system.

Use PyTorch DataLoaders to efficiently batch and process data for an NLP model.