Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[META] Streaming Model/Agent prediction/execution APIs #3328

Open
6 tasks
jngz-es opened this issue Jan 2, 2025 · 1 comment
Open
6 tasks

[META] Streaming Model/Agent prediction/execution APIs #3328

jngz-es opened this issue Jan 2, 2025 · 1 comment
Assignees
Labels
enhancement New feature or request

Comments

@jngz-es
Copy link
Collaborator

jngz-es commented Jan 2, 2025

Is your feature request related to a problem?
The meta issue to track the streaming feature progress.

What solution would you like?

  • Model predict rest api based on transport-reactor-netty4 plugin.
  • Agent execute rest api based on transport-reactor-netty4 plugin.
  • Remote model streaming predict api by supporting sse (sever-sent events) connection with LLMs.
  • MLModelTool streaming run api.
  • Agent streaming execute api with streaming supported tools.
  • Agent streaming execute api with non-streaming tools.

What alternatives have you considered?
A clear and concise description of any alternative solutions or features you've considered.

Do you have any additional context?
Add any other context or screenshots about the feature request here.

@brianf-aws
Copy link
Contributor

Hey @jngz-es is this in progress or do we put on deck?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: On-deck
Development

No branches or pull requests

3 participants