- Changing promts with
the magic wordsdoes affect the result. But I can't say they are always correct. E.G. first question about rbac:
| Search Phrase | Result Quality |
|---|---|
no magic words | just not usable general info |
with the magic words | looks like result is better, but actual configuration suggestions are not correct |
search with AI on our site | most correct answer |
just mentioned Redocly | not usable result |
point to site + additional context | looks correct |
When some product name mensioned, like
API monitoring using Redoclyresults seem to be closer to the actual source, but not examples. Even after usingmagic wordsthe final examples are not correct, e.g. redocly.yaml suggestion for Respect Monitoring:name: API Monitoring on: schedule: - cron: '0 6 * * *' # Runs daily at 6:00 UTC push: paths: - 'openapi.yaml' jobs: lint-api: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - name: Set up Node.js uses: actions/setup-node@v3 with: node-version: '20' - run: npm install - run: redocly lint openapi.yamlIn general, I am not convinced of the positive effects of llms.txt. Results seem pretty similar whether you mention the product name or provide a URL to the docs.
- It was fun to play with different requests. But After first two queries it looks like a repetitive actions with similar outputs.
- When I use
magic wordsthere is some illusion (llm displays Redocly logo when generating answer) that the answer is expected to be correct, but that is not always true. - LLM.txt docs look raw and don't have confirmation that other
big playersare actually going to support it.
| Test Case | Base Query | Magic Words Query | Direct Docs Query | Notes |
|---|---|---|---|---|
| The Gatekeeper | How do I protect this debrief.md page so that it requires someone with the "pearl" role to access it? | Use https://redocly.com/llms.txt and any subsequent links from that index. How do I protect this debrief.md page so that it requires someone with the "pearl" role to access it? | Using https://redocly.com/docs - How do I protect this debrief.md page so that it requires someone with the "pearl" role to access it? | Test RBAC functionality |
| The Storyteller | How do I create a code walkthrough? | Use https://redocly.com/llms.txt and any subsequent links from that index. How do I create a code walkthrough? | Using https://redocly.com/docs - How do I create a code walkthrough? | Test documentation features |
| The Illusionist | How do I only show the mock server in the try it console (not a real server)? | Use https://redocly.com/llms.txt and any subsequent links from that index. How do I only show the mock server in the try it console (not a real server)? | Using https://redocly.com/docs - How do I only show the mock server in the try it console (not a real server)? | Test API console config |
| The Watchman | How do I run API monitoring using Redocly on a regular schedule? | Use https://redocly.com/llms.txt and any subsequent links from that index. How do I run API monitoring using Redocly on a regular schedule? | Using https://redocly.com/docs - How do I run API monitoring using Redocly on a regular schedule? | Test monitoring setup |
| The Artist | How do I make headings purple colored in Redocly? | Use https://redocly.com/llms.txt and any subsequent links from that index. How do I make headings purple colored in Redocly? | Using https://redocly.com/docs - How do I make headings purple colored in Redocly? | Test styling customization |
Rating Scale:
- 1 = Completely wrong/unusable
- 2 = Partially correct but missing key details
- 3 = Generally correct but needs refinement
- 4 = Very good with minor issues
- 5 = Perfect/highly accurate
| Test Case | Base Result Quality | Magic Words Result Quality | Direct Docs Result Quality | Code Examples Accuracy | Best Approach |
|---|---|---|---|---|---|
| The Gatekeeper | 1 | 3 | 4 | Incorrect | Direct |
| The Storyteller | 1 | 3 | 1 | Incorrect | Magic |
| The Illusionist | 2 | 2 | 3 | Incorrect | Direct |
| The Watchman | 1 | 2 | 1 | Incorrect | Magic |
| The Artist | 3 | 2 | 4 | Incorrect | Direct |
links:
| Test Case | Base Result Quality | Magic Words Result Quality | Direct Docs Result Quality | Code Examples Accuracy | Best Approach |
|---|---|---|---|---|---|
| The Gatekeeper | 4 | 2 | 4 | Correct | Base |
| The Storyteller | 5 | 2 | 2 | Correct | Base |
| The Illusionist | 4 | 2 | 2 | Correct | Base |
| The Watchman | 2 | 2 | 4 | Incorrect | Direct |
| The Artist | 5 | 4 | 4 | Correct | Base |
links:
| Test Case | Base Result Quality | Magic Words Result Quality | Direct Docs Result Quality | Code Examples Accuracy | Best Approach |
|---|---|---|---|---|---|
| The Gatekeeper | 1 | 2 | 2 | Incorrect | - |
| The Storyteller | 1 | 1 | 1 | Incorrect | - |
| The Illusionist | 1 | 1 | 1 | Incorrect | - |
| The Watchman | 2 | 3 | 2 | Correct | Magic |
| The Artist | 1 | 2 | 3 | Correct | Base |
links:
| Test Case | Base Result Quality | Magic Words Result Quality | Direct Docs Result Quality | Code Examples Accuracy | Best Approach |
|---|---|---|---|---|---|
| The Gatekeeper | 1 | 1 | 1 | Incorrect | - |
| The Storyteller | 1 | 1 | 1 | Incorrect | - |
| The Illusionist | 3 | 1 | 1 | Incorrect | - |
| The Watchman | 1 | 1 | 3 | Incorrect | Direct |
| The Artist | 3 | 5 | 3 | Correct | Magic |
links:
| Test Case | Base Result Quality | Magic Words Result Quality | Direct Docs Result Quality | Code Examples Accuracy | Best Approach |
|---|---|---|---|---|---|
| The Gatekeeper | 1 | 1 | 1 | Incorrect | - |
| The Storyteller | 1 | 1 | 1 | Incorrect | - |
| The Illusionist | 1 | 1 | 2 | Incorrect | Direct |
| The Watchman | 1 | 1 | 1 | Incorrect | - |
| The Artist | 2 | 2 | 2 | Incorrect | - |
links: Base Queries.md
| Test Case | Base Result Quality | Magic Words Result Quality | Direct Docs Result Quality | Code Examples Accuracy | Best Approach |
|---|---|---|---|---|---|
| The Gatekeeper | 4 | x | 1 | Incorrect | - |
| The Storyteller | 1 | x | 2 | Incorrect | Direct |
| The Illusionist | 2 | x | 3 | - | Direct |
| The Watchman | 3 | x | 1 | - | Base |
| The Artist | 3 | x | 3 | Incorrect | - |
Sonar answer that he don't have access to llms.txt
links:
There is an issue with the editor: Dragging and dropping multiple files randomly renames them.
I think the
llm.txreduces the amount of hallucination, but each LLM seems to handle it differently. Some times this approach also reduce size of text waht LLM outputs so It not really ansewer on question.Sonar prodon't see llm.txt fille at frst run i check it again later and it works this could be issue with our App (downtime)