Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix the parameter to tensor conversion in TRTLLM FastAPI implementation #98

Merged
merged 2 commits into from
Jul 25, 2024

Conversation

tanmayv25
Copy link
Collaborator

The reference of the object was being passed instead of the value.

Before

Max_tokens not respected:

curl -X 'POST'     'http://0.0.0.0:8000/v1/completions'     -H 'accept: application/json'     -H 'Content-Type: application/json'     -d '{
    "model": "opt125m",
    "prompt": "Once upon a time",
    "max_tokens": 16,
    "top_p": 1,
    "n": 1,
    "stream": false,
    "stop": "string",
    "frequency_penalty": 0.0
    }' | jq . 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3254  100  3067  100   187   1525     93  0:00:02  0:00:02 --:--:--  1618
{
  "id": "cmpl-9ba7aa3e-29b9-11ef-ba60-0242ac110008",
  "choices": [
    {
      "finish_reason": null,
      "index": 0,
      "logprobs": null,
      "text": "b' in history, the world will not be the same now. June 10th, 1865 in Portland there was seven thousand Northwest immigrants in Portland against their long history of political oppression with such acts of violence, copious and profound noises and attempts to overpower the Federal Government. These began with a broad capside of a new effort that dissolved the nation and, finally, against liberty.\\n\\nI do not think, or try to make, any claim to the historical antecedents and enormity of this great war. But all I can point for certain is that the flow of terror and disaster was common to all nations. Native Americans, for example, used to burn their houses and uniforms to silence, imprison and imprison their Union and protecting their Nation. They would pursue those people in those sombre murky structures with their vengeance, and, most importantly, they would keep this nation free and secure and let the opposing forces to proceed. This was both a misfortune and an opportunity given to the powerful people who were, in total freedom, to pursue the resource ends of their country, and the genius that was envisaged in 1794.\\n\\n\\xe2\\x80\\x9cAlarms\\xe2\\x80\\x9d of these revolutionary liberators arriving at their barometric bridges rolled on that day. But the continued expressions of rebellion and microwave within miles of the guard stations were endearing. They represented strength, aptitude, courage and trial. They showed that there was no enemy occupying the Nation at all, but an ever- evolving, glorious erection of liberty on the State.\\n\\nThe frustration and misery of all those who had tried in vain to seize power and threaten that liberty stem from the depravity and inhumanity of a past which had so pestered, beat and compelled the people to show defiance and resolution to a process which no man could have anticipated. It is not convenient to discuss these moody recollections, instead, let me just pray for Americans not to believe this world moms with depression, insomnia and despair; for they pleaded that no other Holiday or New Year ever could penetrate the world\\xe2\\x80\\x99s peace, prosperity, freedom and safety and stability and that they were not to be defiant against the New Year.\\n\\nAnd do not base much cynicism upon all these bright and hopeful people who adjusted to the changing times, as we go through a dramatic Spring of 1836. As paranoid enthusiasts, we know that under the full political and moral realms, we stand amid a year of terror war featuring riots, deaths, cruelties \\xe2\\x80\\x94 war, slavery, the war which circulently raged for fifty years upon fifty years from the Battle of Waterloo to Nicaragua, two turbulent years from Great War, bloody Georgia Elections to Pipeline Referendum the greatest of all such tumult of splendor at any date in American history.'"
    }
  ],
  "created": 1718306262,
  "model": "opt125m",
  "system_fingerprint": null,
  "object": "text_completion",
  "usage": null
}

After

Max_tokens respected:

curl -X 'POST'     'http://0.0.0.0:8000/v1/completions'     -H 'accept: application/json'     -H 'Content-Type: application/json'     -d '{
    "model": "opt125m",
    "prompt": "Once upon a time",
    "max_tokens": 16,
    "top_p": 1,
    "n": 1,
    "stream": false,
    "stop": "string",
    "frequency_penalty": 0.0
    }' | jq . 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   472  100   285  100   187    544    357 --:--:-- --:--:-- --:--:--   902
{
  "id": "cmpl-375a897c-29bc-11ef-8045-0242ac110008",
  "choices": [
    {
      "finish_reason": null,
      "index": 0,
      "logprobs": null,
      "text": " in history, the world will not be the same now. June 10th,"
    }
  ],
  "created": 1718307383,
  "model": "opt125m",
  "system_fingerprint": null,
  "object": "text_completion",
  "usage": null
}

Copy link
Contributor

@whoisj whoisj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM :shipit:

@tanmayv25 tanmayv25 merged commit b1653d7 into nnshah1-meetup-04-2024 Jul 25, 2024
3 checks passed
@tanmayv25 tanmayv25 deleted the tanmayv-fastapi-trtllm-fix branch July 25, 2024 19:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants