LangKit, developed by WhyLabs, is an open-source toolkit designed for monitoring Language Learning Models (LLMs). It is engineered to extract relevant signals from both input prompts and output responses, thereby ensuring the safety and security of the language models.
Features of LangKit
- Text Quality Metrics: LangKit provides readability scores, complexity, and grade scores to assess the quality of the text.
- Text Relevance: It offers similarity scores between prompts/responses and against user-defined themes, which helps in understanding the relevance of the text.
- Security and Privacy: LangKit includes features for identifying patterns – count of strings matching a user-defined regex pattern group, jailbreaks – similarity scores with respect to known jailbreak attempts, prompt injection – similarity scores with respect to known prompt injection attacks, and refusals – similarity scores with respect to known LLM refusal of service responses.