Goal:
Develop a semantic chunking (inspired by [3], [4]) Python Library for RAG [2] which uses also an LLM to create chunks and is therefore an advanced method to existing ones. The Python Library is finally accessible via PyPI [1].
What is expected:
Text data from files or URLs are passed to the chunking method which uses a sliding context window with a LLM to create meaningful chunks for RAG. Optimizations like removing chunks with low information content, merging similar chunks or enriching chunks with metadata are encouraged, also with usage of LLMs. The created text chunks can then be stored in a vector database for RAG.
To evaluate the library, simple chunking techniques, containing at least fixed size-, recursive and also semantic chunking, from existing state-of-the-art frameworks (LLamaIndex[5],[6] or LangChain) are used to compare the developed chunking method.
The code, thesis and whole documentation will be hosted public on your own GitHub repository as open source.
Recommended requirements:
Implementation in Python
Access to computer hardware which can run local 4-8 B LLMs
Interest in building LLM AI solutions
Supervisor:
Dr. Marian LUX -
Supervision and thesis in German or English
References:
[2] Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., ... & Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33, 9459-9474.
[3] https://medium.com/the-ai-forum/semantic-chunking-for-rag-f4733025d5f5
[5] https://developers.llamaindex.ai/python/framework/optimizing/basic_strategies/basic_strategies/
[6] https://developers.llamaindex.ai/python/framework/optimizing/basic_strategies/basic_strategies/