Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

llama : add pipeline parallelism support #6017

Merged
merged 23 commits into from
Mar 13, 2024
Merged

llama : add pipeline parallelism support #6017

merged 23 commits into from
Mar 13, 2024

Commits on Mar 12, 2024

  1. llama : add pipeline parallelism support for batch processing with mu… 

    …ltiple CUDA GPUs

ggml-ci
    @slaren
    slaren committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    822121f View commit details
    Browse the repository at this point in the history

  2. server : add -ub, --ubatch-size parameter

    @slaren
    slaren committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    1ac668e View commit details
    Browse the repository at this point in the history

  3. fix server embedding test

    @slaren
    slaren committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    4ddccc2 View commit details
    Browse the repository at this point in the history

  4. llama : fix Mamba inference for pipeline parallelism 

    Tested to work correctly with both `main` and `parallel` examples.
    @compilade
    compilade committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    937966d View commit details
    Browse the repository at this point in the history

  5. llama : limit max batch size to n_batch

    @slaren
    slaren committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    00a415d View commit details
    Browse the repository at this point in the history

  6. add LLAMA_SCHED_MAX_COPIES to configure the number of input copies fo… 

    …r pipeline parallelism

default increase to 4 (from 2)

changing this value may improve performance for some systems, but increases memory usage
    @slaren
    slaren committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    89bfa1f View commit details
    Browse the repository at this point in the history

  7. fix hip build

    @slaren
    slaren committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    aa1e2f8 View commit details
    Browse the repository at this point in the history

  8. Merge remote-tracking branch 'origin/master' into sl/pipeline-paralle… 

    …lism
    @slaren
    slaren committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    deb3e24 View commit details
    Browse the repository at this point in the history

  9. fix sycl build (disable cpy_tensor_async)

    @slaren
    slaren committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    ead5c8b View commit details
    Browse the repository at this point in the history

  10. fix hip build

    @slaren
    slaren committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    255c1ec View commit details
    Browse the repository at this point in the history

Commits on Mar 13, 2024

  1. llama : limit n_batch and n_ubatch to n_ctx during context creation

    @slaren
    slaren committed Mar 13, 2024
    Configuration menu
    Copy the full SHA
    4400153 View commit details
    Browse the repository at this point in the history

  2. llama : fix norm backend

    @slaren
    slaren committed Mar 13, 2024
    Configuration menu
    Copy the full SHA
    9e7cecc View commit details
    Browse the repository at this point in the history

  3. batched-bench : sync after decode

    @ggerganov
    ggerganov committed Mar 13, 2024
    Configuration menu
    Copy the full SHA
    b25a0f1 View commit details
    Browse the repository at this point in the history

  4. swiftui : sync after decode

    @ggerganov
    ggerganov committed Mar 13, 2024
    Configuration menu
    Copy the full SHA
    529e749 View commit details
    Browse the repository at this point in the history

  5. ggml : allow ggml_get_rows to use multiple threads if they are available

    @slaren
    slaren committed Mar 13, 2024
    Configuration menu
    Copy the full SHA
    54cdd47 View commit details
    Browse the repository at this point in the history

  6. check n_ubatch >= n_tokens with non-casual attention

    @slaren
    slaren committed Mar 13, 2024
    Configuration menu
    Copy the full SHA
    cda49d3 View commit details
    Browse the repository at this point in the history

  7. llama : do not limit n_batch to n_ctx with non-casual attn

    @slaren
    slaren committed Mar 13, 2024
    Configuration menu
    Copy the full SHA
    015e1bf View commit details
    Browse the repository at this point in the history

  8. server : construct batch with size of llama_n_batch

    @ggerganov
    ggerganov committed Mar 13, 2024
    Configuration menu
    Copy the full SHA
    0d934ee View commit details
    Browse the repository at this point in the history

  9. ggml_backend_cpu_graph_compute : fix return value when alloc fails

    @slaren
    slaren committed Mar 13, 2024
    Configuration menu
    Copy the full SHA
    3c38789 View commit details
    Browse the repository at this point in the history

  10. llama : better n_batch and n_ubatch comment

    @slaren
    slaren committed Mar 13, 2024
    Configuration menu
    Copy the full SHA
    9092883 View commit details
    Browse the repository at this point in the history

  11. fix merge

    @slaren
    slaren committed Mar 13, 2024
    Configuration menu
    Copy the full SHA
    cb580a6 View commit details
    Browse the repository at this point in the history

  12. small fix

    @slaren
    slaren committed Mar 13, 2024
    Configuration menu
    Copy the full SHA
    1f56481 View commit details
    Browse the repository at this point in the history

  13. reduce default n_batch to 2048

    @slaren
    slaren committed Mar 13, 2024
    Configuration menu
    Copy the full SHA
    976176d View commit details
    Browse the repository at this point in the history