Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

inconsistency in titles generated by GPT #fixed #1137

Open
wants to merge 2 commits into
base: develop
Choose a base branch
from
Open

inconsistency in titles generated by GPT #fixed #1137

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
0b908d8
commit this
alisha2501 Jul 2, 2024
53ccb98
inconsistency in titles generated by GPT #requestchanges
alisha2501 Jul 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 32 additions & 2 deletions ...ion/src/main/java/org/togetherjava/tjbot/features/moderation/TransferQuestionCommand.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
package org.togetherjava.tjbot.features.moderation;


import net.dv8tion.jda.api.JDAConstants;
import net.dv8tion.jda.api.EmbedBuilder;
import net.dv8tion.jda.api.JDA;
import net.dv8tion.jda.api.entities.Guild;
Expand Down Expand Up @@ -84,6 +86,32 @@ public TransferQuestionCommand(Config config, ChatGptService chatGptService) {
this.chatGptService = chatGptService;
}


public class TitleGenerator {

// Define regex patterns for leading and trailing quotation marks
private static final Pattern LEADING_TRAILING_QUOTES = Pattern.compile("^['\"]|['\"]$");

public static String generateTitle(String originalMessage, ChatGptService chatGptService) {
String chatGptPrompt =
"Summarize the following text into a concise title or heading not more than 4-5 words, remove quotations if any: %s"
.formatted(originalMessage);
Optional<String> chatGptResponse = chatGptService.ask(chatGptPrompt, "");
String title = chatGptResponse.orElse(createTitle(originalMessage));
title = LEADING_TRAILING_QUOTES.matcher(title).replaceAll("");
Copy link
Contributor

@surajkumar surajkumar Jul 2, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Problem is that you're also removing intentional quotations.

Consider the following example:
"hello" is being printed but not "hello world" becomes hello" is being printed but not "hello world

'hello world" this is also now just becoming hello world

You should probably let ChatGPT handle this entirely and update the prompt to ensure that and test it does it the way we intend it to.

Copy link
Contributor

@surajkumar surajkumar Jul 2, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In sense, you're really just trying the following on-top of ChatGPT:

if(str.startsWith("\"") && str.endsWith("\"")) {
  str = str.substring(1, str.length - 1);
}
if(str.startsWith("'") && str.endsWith("'")) {
  str = str.substring(1, str.length - 1);
}

but with uneeded complexity.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We tried to deal at prompt level, unfortunately GPT is very inconsistent and sometimes would end up adding quotations regardless. The best possible way forward is to let GPT do it's thing and then like suraj mentioned look for titles that starting with quotation marks and only remove those that way we handle the edge case.

if (title.length() > JDAConstants.MAX_TITLE_LENGTH) {
title = title.substring(0, JDAConstants.MAX_TITLE_LENGTH);
}

return title;
}

private static String createTitle(String originalMessage) {
return originalMessage.substring(0,
Math.min(originalMessage.length(), JDAConstants.MAX_TITLE_LENGTH));
}
}

@Override
public void onMessageContext(MessageContextInteractionEvent event) {

Expand All @@ -96,15 +124,18 @@ public void onMessageContext(MessageContextInteractionEvent event) {
String originalChannelId = event.getTarget().getChannel().getId();
String authorId = event.getTarget().getAuthor().getId();
String mostCommonTag = tags.getFirst();

String chatGptPrompt =
"Summarize the following text into a concise title or heading not more than 4-5 words, remove quotations if any: %s"
.formatted(originalMessage);
Optional<String> chatGptResponse = chatGptService.ask(chatGptPrompt, "");
String title = chatGptResponse.orElse(createTitle(originalMessage));
title = title.replaceAll("^\"|\"$", "").replaceAll("^'|'$", "");
alisha2501 marked this conversation as resolved.
Show resolved Hide resolved
alisha2501 marked this conversation as resolved.
Show resolved Hide resolved
if (title.length() > TITLE_MAX_LENGTH) {
title = title.substring(0, TITLE_MAX_LENGTH);
}


TextInput modalTitle = TextInput.create(MODAL_TITLE_ID, "Title", TextInputStyle.SHORT)
.setMaxLength(TITLE_MAX_LENGTH)
.setMinLength(TITLE_MIN_LENGTH)
Expand Down Expand Up @@ -176,8 +207,7 @@ private void transferFlow(ModalInteractionEvent event, String channelId, String
.retrieveUserById(authorId)
.flatMap(fetchedUser -> createForumPost(event, fetchedUser))
.flatMap(createdForumPost -> dmUser(event.getChannel(), createdForumPost,
event.getGuild())
.and(sendMessageToTransferrer.apply(createdForumPost)))
event.getGuild()).and(sendMessageToTransferrer.apply(createdForumPost)))
.flatMap(dmSent -> deleteOriginalMessage(event.getJDA(), channelId, messageId))
.queue();
}
Expand Down
Loading