Human Annotation for LLM apps
Collaborate with your team and add scores via the Langfuse UI. You can add scores to both traces and observations within a trace.
Common use cases:
- Collaboration: Enable team collaboration by inviting other internal members to annotate a subset of traces and observations. This manual evaluation can enhance the overall accuracy and reliability of your results by incorporating diverse perspectives and expertise.
- Annotation data consistency: Create score configurations for annotation workflows to ensure that all team members are using standardized scoring criteria. Hereby configure categorical, numerical or binary score types to capture different aspects of your data.
- Evaluation of new product features: This feature can be useful for new use cases where no other scores have been allocated yet.
- Benchmarking of other scores: Establish a human baseline score that can be used as a benchmark to compare and evaluate other scores. This can provide a clear standard of reference and enhance the objectivity of your performance evaluations.
Create score configurations
To use annotation, you need to create a score configuration. You can create multiple score configurations for different types of scores. Score configurations are immutable. However, you can archive configs if you no longer want to use them in annotation. Archived configs can be restored at any time.
To create a score configuration:
- Navigate to
Settings, locate theScore Configstable, and clickAdd new score config. - Specify the name of the score configuration and the type of score you want to create. You can choose between
Categorical,Numeric, andBooleanscore types. - Optionally, add a description to provide additional context for your team members.
Your configs are now available for annotation of traces and observations.
Data Labelling on LLM traces or observations
To annotate a trace or observation:
- Navigate to the trace detail view.
- Click on the
Annotatebutton. - Select the scores you want to add.
- Fill in the score values. The scores will be saved automatically. Annotation scores can be edited or deleted at any time.
- Optionally, add comments to individual scores.
View scores on trace or observation
Upon completing annotation click on the Scores tab to view a table of all the scores that have been added to the trace or observation.
