llm-serving

Here are 29 public repositories matching this topic...

ray-project / ray

Ray is a unified framework for scaling AI and Python applications. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

Updated Dec 13, 2023
Python

vllm-project / vllm

Star

A high-throughput and memory-efficient inference and serving engine for LLMs

inference pytorch transformer llama gpt model-serving mlops llm llmops llm-serving

Updated Dec 13, 2023
Python

bentoml / OpenLLM

Star

Operating LLMs in production

Updated Dec 12, 2023
Python

skypilot-org / skypilot

Star

SkyPilot: Run LLMs, AI, and Batch jobs on any cloud. Get maximum savings, highest GPU availability, and managed execution—all with a simple interface.

Updated Dec 13, 2023
Python

liguodongiot / llm-action

Star

本项目旨在分享大模型相关技术原理以及实战经验。

llm llmops llm-serving llm-training llm-inference

Updated Dec 11, 2023
Jupyter Notebook

ray-project / ray-llm

Star

RayLLM - LLMs on Ray

distributed-systems transformers ray serving large-language-models llm llmops llm-serving llm-inference

Updated Dec 6, 2023
Python

mosecorg / mosec

Star

A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine

python rust machine-learning deep-learning mxnet tensorflow gpu cv pytorch tts hacktoberfest model-serving nerual-network machine-learning-platform jax mlops llm llm-serving

Updated Dec 10, 2023
Python

predibase / lorax

Star

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

transformers pytorch llama gpt lora model-serving fine-tuning llm llmops llm-serving llm-inference

Updated Dec 13, 2023
Python

ray-project / ray-educational-materials

Star

This is suite of the hands-on training materials that shows how to scale CV, NLP, time-series forecasting workloads with Ray.

deep-learning ray distributed-machine-learning ray-tune ray-train ray-distributed llm generative-ai ray-serve ray-data llm-serving llm-inference

Updated Dec 4, 2023
Jupyter Notebook

substratusai / runbooks

Star

Finetune LLMs on K8s by using Runbooks

kubernetes kubernetes-operator mlops ml-platform llmops llm-serving llm-training llm-inference

Updated Nov 21, 2023
Go

chenhunghan / ialacol

Star

🪶 Lightweight OpenAI drop-in replacement for Kubernetes

python kubernetes ai gpu helm cuda openai cloudnative llm langchain llm-serving llamacpp ggml gptq llm-inference

Updated Nov 28, 2023
Python

sugarcane-ai / sugarcane-ai

Star

Open Source Framework to build, train and monetise cross LLM and high accuracy Prompt Packages powered by Micro LLMs

performance framework ai packages next prompts llm prompt-engineering llmops llm-serving llm-chain llm-training llm-framework llm-finetuning

Updated Dec 12, 2023
TypeScript

ray-project / llms-in-prod-workshop-2023

Star

Deploy and Scale LLM-based applications

ray anyscale llm llms llm-serving llm-inference

Updated Jun 15, 2023
Jupyter Notebook

mani-kantap / llm-inference-solutions

Star

A collection of all available inference solutions for the LLMs

llmops llm-serving llm-inference

Updated Nov 19, 2023

friendliai / periflow-client

Star

PeriFlow: the fastest serving engine for generative AI such as LLMs

ai ml inference mpt gpt inference-server mistral inference-engine serving mlops gpt3 llm llms generative-ai llmops llm-serving llm-inference llama2 llm-ops

Updated Dec 7, 2023
Python

sugarcane-ai / sugarcane-ai.github.io

Star

training api workflow microservices framework typescript architecture prompt reusable prompts fine-tuning llm prompt-engineering generative-ai llmops llm-serving llm-training

Updated Dec 11, 2023
Astro

oscinis-com / Awesome-LLM-Productization

Star

Awesome-LLM-Productization: a curated list of tools/tricks/news/regulations about AI and Large Language Model (LLM) productization

machine-learning awesome ai production transformer awesome-list mlops large-language-models llm llmops llm-serving llm-inference

Updated Oct 10, 2023

rohan-paul / LLM-FineTuning-Large-Language-Models

Star

LLM (Large Language Model) FineTuning

pytorch gpt-3 large-language-models llm llm-serving gpt3-turbo llm-training llm-inference open-source-llm llama2 llm-finetuning mistral-7b

Updated Dec 12, 2023
Jupyter Notebook

ray-project / anyscale-berkeley-ai-hackathon

Star

Ray and Anyscale for UC Berkeley AI Hackathon!

hackathon berkeley-ai ray-distributed anyscale llm llm-serving llm-inference

Updated Jun 17, 2023
Jupyter Notebook

A self-hosted personal chatbot API with FastAPI. It allows you to interact with the Llama2 LLM (and other open-source LLMs) to have natural language conversations, generate text, and perform various language-related tasks.

langchain llm-serving llamacpp llama2

Updated Nov 5, 2023
Jupyter Notebook

Improve this page

Add a description, image, and links to the llm-serving topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the llm-serving topic, visit your repo's landing page and select "manage topics."

Learn more

Nov	DEC	Jan
	13
2022	2023	2024

llm-serving

Here are 29 public repositories matching this topic...

ray-project / ray

vllm-project / vllm

bentoml / OpenLLM

skypilot-org / skypilot

liguodongiot / llm-action

ray-project / ray-llm

mosecorg / mosec

predibase / lorax

ray-project / ray-educational-materials

substratusai / runbooks

chenhunghan / ialacol

sugarcane-ai / sugarcane-ai

ray-project / llms-in-prod-workshop-2023

mani-kantap / llm-inference-solutions

friendliai / periflow-client

sugarcane-ai / sugarcane-ai.github.io

oscinis-com / Awesome-LLM-Productization

rohan-paul / LLM-FineTuning-Large-Language-Models

ray-project / anyscale-berkeley-ai-hackathon

ehsanghaffar / ein-llm

Improve this page

Add this topic to your repo