New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

缓存C<sup>KV</sup><sub>t</sub> 多卡并行推理是否需要每张卡缓存一份 #41

Open

c-dafan opened this issue May 20, 2024 · 0 comments

c-dafan commented May 20, 2024

缓存C^KV_t在推理时，是否需要重新计算k^C_t，v^C_t？如果需要，在多卡推理的时候，每张卡需要完整的C^KV_t，这样需要存储多份吧

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment