Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

set summarizer write to tempfile #330

Merged
merged 2 commits into from
Apr 24, 2024
Merged

set summarizer write to tempfile #330

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
8f96b15
set summarizer write to /tmp
MichaelClifford Apr 24, 2024
9f8dab0
use tempfile for summarizer
MichaelClifford Apr 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion recipes/natural_language_processing/summarizer/app/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
langchain
langchain_openai
streamlit
streamlit
pypdf
7 changes: 4 additions & 3 deletions recipes/natural_language_processing/summarizer/app/summarizer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
from langchain_community.callbacks import StreamlitCallbackHandler
from langchain_community.document_loaders import PyPDFLoader
import streamlit as st
import tempfile
import requests
import time
import os
Expand Down Expand Up @@ -49,12 +50,12 @@ def read_file(file):
file_type = file.type

if file_type == "application/pdf":
with open(file.name, "wb") as f:
temp = tempfile.NamedTemporaryFile()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You still need to remove this when you are done with it?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I tested it the tempfile deleted after the file was no longer being used.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this some magic in Python?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

Use the [tempfile module](http://docs.python.org/2/library/tempfile.html); it creates temporary files that auto-delete.

From the [tempfile.NamedTemporaryFile() documentation](http://docs.python.org/2/library/tempfile.html#tempfile.NamedTemporaryFile):

    If delete is true (the default), the file is deleted as soon as it is closed.

Cool

with open(temp.name, "wb") as f:
f.write(file.getvalue())
loader = PyPDFLoader(file.name)
loader = PyPDFLoader(temp.name)
pages = loader.load()
text = "".join([p.page_content for p in pages])
os.remove(file.name)

if file_type == "text/plain":
text = file.read().decode()
Expand Down