Back to Blog

GMI Cloud ?? Tensormesh ?? 4 ? LLM ????

2026-01-26

  • Yihua Cheng

??? SSD ??? KVCache ????????????? 4 ???????????

?? GMI Cloud ? Tensormesh ??????????????????

????????

  • ??? SSD ??? KVCache ????????? Token ???Time to First Token, TTFT???? 4 ??
  • ???????? 3% ????? 50%?????????????????????????
  • ??????????????????????????????? AI ?????????????
  • ?????? KV ???????????? GPU ??????????????????????????

??????????? KV Cache ?????????? LLM ??????????????????????? AI ?????????????????????????????????????????

????????

1. ??????????????

???????????????????????????GMI ??????????????????????????????????????????????????????

???????

  • ???????? AI ???????????? 1 ?? 10 ??????
  • ????????????????????????????????????????????????????
  • ??????????????????????????????????????
  • ??????????????????????????????????????????????????????

??????? KV Cache ????????????????????????

2. ????

????????????

  • ?? vLLM????? KVCache ???
  • ?? LMCache ? vLLM???? Tensormesh ?????

????????????

  • ???????KVCache?????????????
  • ?? + ??? SSD ??????????? KV Cache ???????

???????????????????????????????

3. ????

??????? KVCache ?????

  • ????? 1.4 ?? TTFT ???
  • ?????????KV Cache ???????????????????????

SSD ???KVCache???

  • ??? 4 ?? TTFT ???
  • ??????????????????? 50%?
  • ??????????????????

???????????? KV ???????????????????

????????????? SSD ??????????? KVCache ???????????????????

4. ????

? Token ???TTFT?
TTFT——????????????——?????????????? 4 ???????????????????

Bar graph comparing average TTFT (Total Time for Task) in seconds for three configurations: vLLM with LMCache CPU + Disk (0.331s), vLLM with LMCache CPU offloading (0.8148s), and Native vLLM (1.1629s).

?? KV Cache ???
??????????????KV Cache ???????????????????? 50%????????????????????

Bar graph showing average prefix cache hit rate percentages for three configurations: Native vLLM (3.43%), vLLM with LMCache CPU offloading (23.84%), and vLLM with LMCache CPU and Disk (53.21%).

????????
?? TTFT ??????????????????????????????????????????????????????????

Line graph displaying TTFT (Time To First Token) over time, comparing three models: vLLM with LMCache CPU + Disk, vLLM with LMCache CPU offloading, and Native vLLM. The y-axis represents TTFT in seconds, while the x-axis shows time in a specific format.

KV Cache ? LLM ???????

???????? GMI ??????????????????????????????

  • ???????????? KV ??????????? GPU ????
  • ???????????????AI ?????????????
  • ?????????????????????? QPS?
  • ?????????SSD ?????????????? RAM ???

?????GMI Cloud ??????????????????????“???????????????”??????????????????????????????????????

?????????????

  • ???????????????????????????
  • ??????RAM + SSD??????????????
  • LMCache ???????????????? vLLM ??????
  • ?????????????AI ??????? LLM ?????????????

Tensormesh ??

Tensormesh ??? AI ???????????????????????????? AI????????????????????????? GPU ??????? 10 ??????????????????????????Tensormesh ?????????????????? Laude Ventures ??? 450 ?????????

GMI Cloud ??

GMI Cloud ?????? GPU ??????????????? AI ???????????????? NVIDIA ??????GMI ????? GPU ????? NVIDIA Blackwell ?? H100 ? H200 GPU ??????GMI ???????????????????????? AI ????????????????

Table of Contents

Resources:

Share via:

Join the conversation on GitHub Discussions or Slack.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

More from the blog

Tech Explained

2026-05-04

Deepseek V4 explained, and why it matters to your wallet

Read Article

lmcache

2026-04-28

Stop Calling It KV Cache: It’s Something Much Bigger

Read Article

Benchmark

2026-04-22

LMCache on Amazon SageMaker HyperPod: Accelerating LLM Inference with Managed Tiered KV Cache

Read Article
All Articles
GitHub Repo
Join us via: