-
Notifications
You must be signed in to change notification settings - Fork 43k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. Weβll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(forge/llm): Add LlamafileProvider
#7091
base: master
Are you sure you want to change the base?
feat(forge/llm): Add LlamafileProvider
#7091
Conversation
β¦der for llamafiles. Currently it just extends OpenAIProvider and only overrides methods that are necessary to get the system to work at a basic level. Update ModelProviderName schema and config/configurator so that app startup using this provider is handled correctly. Add 'mistral-7b-instruct-v0' to OpenAIModelName/OPEN_AI_CHAT_MODELS registries.
β¦-Instruct chat template, which supports the 'user' & 'assistant' roles but does not support the 'system' role.
β¦kens`, and `get_tokenizer` from classmethods so I can override them in LlamafileProvide (and so I can access instance instance attributes from inside them). Implement class `LlamafileTokenizer` that calls the llamafile server's `/tokenize` API endpoint.
β¦tes on the integration; add helper scripts for downloading/running a llamafile + example env file.
β¦gs for reproducibility
β¦ange serve.sh to use model's full context size (this does not seem to cause OOM errors, surpisingly).
β Deploy Preview for auto-gpt-docs canceled.
|
This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request. |
@CodiumAI-Agent /review |
PR Review
Code feedback:
β¨ Review tool usage guide:Overview: The tool can be triggered automatically every time a new PR is opened, or can be invoked manually by commenting on any PR.
See the review usage page for a comprehensive guide on using this tool. |
@k8si any chance you could enable maintainer write access on this PR? |
@Pwuts it doesn't look like I have the ability to do that. I added you as a maintainer to the forked project, is that sufficient or do others need write access? Alternatively, you could branch off my branch and I can just accept the changes via PR? |
Conflicts have been resolved! π A maintainer will review the pull request shortly. |
β¦vider`, `GroqProvider` and `LlamafileProvider` and rebase the latter three on `BaseOpenAIProvider`
Conflicts have been resolved! π A maintainer will review the pull request shortly. |
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #7091 +/- ##
==========================================
+ Coverage 42.64% 44.83% +2.19%
==========================================
Files 82 65 -17
Lines 4772 3669 -1103
Branches 655 497 -158
==========================================
- Hits 2035 1645 -390
+ Misses 2645 1959 -686
+ Partials 92 65 -27
Flags with carried forward coverage won't be shown. Click here to find out more. β View full report in Codecov by Sentry. |
This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request. |
Conflicts have been resolved! π A maintainer will review the pull request shortly. |
Background
This draft PR is a step toward enabling the use of local models in AutoGPT by adding llamafile as an LLM provider.
Related issues:
Changes ποΈ
For full documentation of the changes involved in this PR, as well as instructions for running AutoGPT with the llamafile backend, see
autogpts/autogpt/llamafile-integration/README.md
.Summary of changes:
LlamafileProvider
, a newChatModelProvider
for llamafiles. Currently it just extendsOpenAIProvider
and only overrides methods that are necessary to get the system to work at a basic level.mistral-7b-instruct-v0
toOpenAIModelName
/OPEN_AI_CHAT_MODELS
registries. This is the only model currently supported byLlamafileProvider
because this is the only model I tested anything with.LLM_PROVIDER
that, when set to 'llamafile', will useLllamafileProvider
in agents rather thanOpenAIProvider
.Limitations:
PR Quality Scorecard β¨
+2 pts
+5 pts
+5 pts
+5 pts
-4 pts
+4 pts
+5 pts
-5 pts
agbenchmark
to verify that these changes do not regress performance? β+10 pts