SttTtsPlugin error when launching space #2206

y4my4my4m · 2025-01-12T15:48:22Z

Describe the bug

[Space] Initialized => https://x.com/i/broadcasts/XXXXXXXXXXXXXXX
 ["◎ [SttTtsPlugin] init => Space fully ready. Subscribing to events."] 

 ◎ LOGS
   [SttTtsPlugin] Plugin config => 
   ⛔ ERRORS
     [Space] Error launching Space => 
     {} 
   ⛔ ERRORS
     [Space] Error in routine => 
     {}

To Reproduce

Current commit: d55c86c

    "twitterSpaces": {
        "maxSpeakers": 2,
        "topics": [
            "Recent News",
        ],
        "typicalDurationMinutes": 45,
        "idleKickTimeoutMs": 300000,
        "minIntervalBetweenSpacesMinutes": 1,
        "businessHoursOnly": false,
        "randomChance": 1,
        "enableIdleMonitor": true,
        "enableSttTts": true,
        "enableRecording": false,
        "voiceId": "XYZ",
        "sttLanguage": "en",
        "gptModel": "gpt-4o-mini",
        "elevenLabsModel": "eleven_flash_v2_5",
        "systemPrompt": "Roleplay as a clown",
        "speakerMaxDurationMs": 120000
    },

Expected behavior

Launches the space without SttTtsPlugin crashing

The text was updated successfully, but these errors were encountered:

y4my4my4m · 2025-01-12T17:20:21Z

This is crashing it:

eliza/packages/client-twitter/src/plugins/SttTtsSpacesPlugin.ts

Line 104 in d55c86c

elizaLogger.log("[SttTtsPlugin] Plugin config =>", config);

Removing that log fixes it..im assuming something's off with the config?

daniellinuk · 2025-01-14T17:32:36Z

Hi @y4my4my4m

I commented out the log at line 104 as suggested but still getting the same error. Any new findings?

I am able to see the agent launched a space but keeps having connection issues.

y4my4my4m · 2025-01-14T21:12:10Z

Hi @y4my4my4m

I commented out the log at line 104 as suggested but still getting the same error. Any new findings?

I am able to see the agent launched a space but keeps having connection issues.

Did you correctly build after editing?
Any diffs with your .env.sample and your .env?

daniellinuk · 2025-01-15T04:17:20Z

Hi @y4my4my4m
I commented out the log at line 104 as suggested but still getting the same error. Any new findings?
I am able to see the agent launched a space but keeps having connection issues.

Did you correctly build after editing? Any diffs with your .env.sample and your .env?

Works after build, thanks.

CactusByte · 2025-01-16T15:52:29Z

I'm having this error but the agent keep crashing after removing log and building

Error: ICE connection failed
at RTCPeerConnection. (file:///Users/user/Documents/Superman/eliza/node_modules/agent-twitter-client/dist/node/esm/index.mjs:4476:28)
at /Users/user/Documents/Superman/eliza/node_modules/@roamhq/wrtc/lib/eventtarget.js:37:18
at Set.forEach ()
at /Users/user/Documents/Superman/eliza/node_modules/@roamhq/wrtc/lib/eventtarget.js:33:15
at process.processTicksAndRejections (node:internal/process/task_queues:85:11)
Emitted 'error' event on JanusClient instance at:
at RTCPeerConnection. (file:///Users/user/Documents/Superman/eliza/node_modules/agent-twitter-client/dist/node/esm/index.mjs:4476:14)
at /Users/user/Documents/Superman/eliza/node_modules/@roamhq/wrtc/lib/eventtarget.js:37:18
[... lines matching original stack trace ...]
at process.processTicksAndRejections (node:internal/process/task_queues:85:11)

[JanusClient] createRoom => {"janus":"success","session_id":439686258888454,"transaction":"7bdmxunc","sender":3795148812314350,"plugindata":{"plugin":"janus.plugin.videoroom","data":{"videoroom":"created","room":"1YpJklwYEPPxj","permanent":false}}}
[JanusClient] Room '1YpJklwYEPPxj' created successfully
[JanusClient] joined room => publisherId= 5493541486620820
[JanusClient] createOffer...
[JanusClient] sending configure with JSEP...
[JanusClient] waiting for answer...
[JanusClient] Initialization complete
[Space] Publishing broadcast...
[Space] Connecting chat...
[ChatClient] Connecting => wss://prod-chatman-ancillary-us-east-1.pscp.tv/chatapi/v1/chatnow
[JanusClient] got answer => setRemoteDescription
[ChatClient] Connected
[Space] Initialized => https://x.com/i/broadcasts/1YpJklwYEPPxj
["◎ [SttTtsPlugin] init => Space fully ready. Subscribing to events."]

◎ LOGS
[SttTtsPlugin] Plugin config =>
⛔ ERRORS
[Space] Error launching Space =>
{}
⛔ ERRORS
[Space] Error in routine =>
{}
[JanusClient] webrtcup => 3795148812314350
/Users/User/Documents/Superman/eliza/node_modules/@primuslabs/zktls-core-sdk/dist/algorithm/client_plugin.js:1

y4my4my4m · 2025-01-18T02:14:21Z

@CactusByte did you build?

tcm390 · 2025-01-18T03:43:23Z

Describe the bug

[Space] Initialized => https://x.com/i/broadcasts/XXXXXXXXXXXXXXX
 ["◎ [SttTtsPlugin] init => Space fully ready. Subscribing to events."] 

 ◎ LOGS
   [SttTtsPlugin] Plugin config => 
   ⛔ ERRORS
     [Space] Error launching Space => 
     {} 
   ⛔ ERRORS
     [Space] Error in routine => 
     {}

To Reproduce

Current commit: d55c86c

    "twitterSpaces": {
        "maxSpeakers": 2,
        "topics": [
            "Recent News",
        ],
        "typicalDurationMinutes": 45,
        "idleKickTimeoutMs": 300000,
        "minIntervalBetweenSpacesMinutes": 1,
        "businessHoursOnly": false,
        "randomChance": 1,
        "enableIdleMonitor": true,
        "enableSttTts": true,
        "enableRecording": false,
        "voiceId": "XYZ",
        "sttLanguage": "en",
        "gptModel": "gpt-4o-mini",
        "elevenLabsModel": "eleven_flash_v2_5",
        "systemPrompt": "Roleplay as a clown",
        "speakerMaxDurationMs": 120000
    },

Expected behavior

Launches the space without SttTtsPlugin crashing

Hi @y4my4my4m, could you please provide steps to reproduce the issue? Could you check if the issue persists on the latest develop branch? I wasn’t able to reproduce it on my end. Thank you!

daniellinuk · 2025-01-18T05:13:39Z

I'm having this error but the agent keep crashing after removing log and building

Error: ICE connection failed at RTCPeerConnection. (file:///Users/user/Documents/Superman/eliza/node_modules/agent-twitter-client/dist/node/esm/index.mjs:4476:28) at /Users/user/Documents/Superman/eliza/node_modules/@roamhq/wrtc/lib/eventtarget.js:37:18 at Set.forEach () at /Users/user/Documents/Superman/eliza/node_modules/@roamhq/wrtc/lib/eventtarget.js:33:15 at process.processTicksAndRejections (node:internal/process/task_queues:85:11) Emitted 'error' event on JanusClient instance at: at RTCPeerConnection. (file:///Users/user/Documents/Superman/eliza/node_modules/agent-twitter-client/dist/node/esm/index.mjs:4476:14) at /Users/user/Documents/Superman/eliza/node_modules/@roamhq/wrtc/lib/eventtarget.js:37:18 [... lines matching original stack trace ...] at process.processTicksAndRejections (node:internal/process/task_queues:85:11)

[JanusClient] createRoom => {"janus":"success","session_id":439686258888454,"transaction":"7bdmxunc","sender":3795148812314350,"plugindata":{"plugin":"janus.plugin.videoroom","data":{"videoroom":"created","room":"1YpJklwYEPPxj","permanent":false}}} [JanusClient] Room '1YpJklwYEPPxj' created successfully [JanusClient] joined room => publisherId= 5493541486620820 [JanusClient] createOffer... [JanusClient] sending configure with JSEP... [JanusClient] waiting for answer... [JanusClient] Initialization complete [Space] Publishing broadcast... [Space] Connecting chat... [ChatClient] Connecting => wss://prod-chatman-ancillary-us-east-1.pscp.tv/chatapi/v1/chatnow [JanusClient] got answer => setRemoteDescription [ChatClient] Connected [Space] Initialized => https://x.com/i/broadcasts/1YpJklwYEPPxj ["◎ [SttTtsPlugin] init => Space fully ready. Subscribing to events."]

◎ LOGS [SttTtsPlugin] Plugin config => ⛔ ERRORS [Space] Error launching Space => {} ⛔ ERRORS [Space] Error in routine => {} [JanusClient] webrtcup => 3795148812314350 /Users/User/Documents/Superman/eliza/node_modules/@primuslabs/zktls-core-sdk/dist/algorithm/client_plugin.js:1

@CactusByte I have been here before, it could be one of the things below.

what's your ubuntu version? needs to be at least 22. See this issue Error: Could not find wrtc binary on any of the paths when running pnpm start #1965 this is related to @roamhq node module
could be you need to install FFmpeg
could be the SttTts plugin is not able to read either the elevenlabs or openai API keys

suggest you add more error logs for startSpace function in spaces.ts

y4my4my4m · 2025-01-19T13:04:10Z

@daniellinuk

im on arch, not ubuntu, but im not seeing any wrtc binary issues
ffmpeg is installed

one thing that is wrong with the spaces plugin is that it's reading the exported OPEN_API_KEY value from my zshrc instead of what's defined in the .env:

 // Retrieve keys
 const openAiKey = process.env.OPENAI_API_KEY || "";
 const elevenLabsKey = process.env.ELEVENLABS_XI_API_KEY || "";

should be like:

 runtime.getSetting("OPENAI_API_KEY") ??

instead.

However that is not the issue as it works when i just comment out the config.

@tcm390

I can't seem to run the latest dev branch, but on the latest release v0.1.8+build.1 i am getting the same error on a fresh install.

node v23.3.0

tcm390 · 2025-01-19T23:16:19Z

@daniellinuk

im on arch, not ubuntu, but im not seeing any wrtc binary issues

ffmpeg is installed

one thing that is wrong with the spaces plugin is that it's reading the exported OPEN_API_KEY value from my zshrc instead of what's defined in the .env:

// Retrieve keys
const openAiKey = process.env.OPENAI_API_KEY || "";
const elevenLabsKey = process.env.ELEVENLABS_XI_API_KEY || "";
should be like:

runtime.getSetting("OPENAI_API_KEY") ??
instead.

However that is not the issue as it works when i just comment out the config.

@tcm390

I can't seem to run the latest dev branch, but on the latest release v0.1.8+build.1 i am getting the same error on a fresh install.

node v23.3.0

Got it thanks for clarifying 🫶 will check!

tcm390 · 2025-01-20T23:06:58Z

someone reported this:

It doesn't work for me, i'm constantly having this kind of error which crashes the agent. I've been trying to solve the issue for a couple of days without finding a fix.

i noticed it's the same issue already described here:
#2206

Error message:

Error: ICE connection failed
at RTCPeerConnection. (file:///C:/eliza/node_modules/agent-twitter-client/dist/node/esm/index.mjs:4476:28)
at C:\eliza\node_modules@roamhq\wrtc\lib\eventtarget.js:37:18
at Set.forEach ()
at C:\eliza\node_modules@roamhq\wrtc\lib\eventtarget.js:33:15
at process.processTicksAndRejections (node:internal/process/task_queues:85:11)
Emitted 'error' event on JanusClient instance at:
at RTCPeerConnection. (file:///C:/eliza/node_modules/agent-twitter-client/dist/node/esm/index.mjs:4476:14)
at C:\eliza\node_modules@roamhq\wrtc\lib\eventtarget.js:37:18
[... lines matching original stack trace ...]
at process.processTicksAndRejections (node:internal/process/task_queues:85:11)

Node.js v23.3.0
C:\eliza\agent:
ERR_PNPM_RECURSIVE_RUN_FIRST_FAIL @elizaos/[email protected] start: node --loader ts-node/esm src/index.ts "--isRoot" "--characters=eliza/characters/character.character.json"
Exit status 7
ELIFECYCLE Command failed with exit code 7.

y4my4my4m · 2025-01-21T04:25:01Z

@tcm390 Im not getting any ICE issues.

worksgoodcompany · 2025-01-21T15:26:10Z

@tcm390 Im not getting any ICE issues.

I'll try with the latest updates and let you know if the problem persists. But is there any specific requirement or tip i should consider to make it run correctly? What's your dev env? are you using the devcontainer? I am using a windows host and wsl.

worksgoodcompany · 2025-01-21T17:26:59Z

@tcm390 Im not getting any ICE issues.

I'll try with the latest updates and let you know if the problem persists. But is there any specific requirement or tip i should consider to make it run correctly? What's your dev env? are you using the devcontainer? I am using a windows host and wsl.

Still getting same error:

Error: ICE connection failed
at RTCPeerConnection. (file:///workspace/node_modules/agent-twitter-client/dist/node/esm/index.mjs:4476:28)
at /workspace/node_modules/@roamhq/wrtc/lib/eventtarget.js:37:18
at Set.forEach ()
at /workspace/node_modules/@roamhq/wrtc/lib/eventtarget.js:33:15
at process.processTicksAndRejections (node:internal/process/task_queues:85:11)
Emitted 'error' event on JanusClient instance at:
at RTCPeerConnection. (file:///workspace/node_modules/agent-twitter-client/dist/node/esm/index.mjs:4476:14)
at /workspace/node_modules/@roamhq/wrtc/lib/eventtarget.js:37:18
[... lines matching original stack trace ...]
at process.processTicksAndRejections (node:internal/process/task_queues:85:11)

Node.js v23.5.0
/workspace/agent:
ERR_PNPM_RECURSIVE_RUN_FIRST_FAIL @elizaos/[email protected] start: node --loader ts-node/esm src/index.ts "--isRoot" "--characters=eliza/characters/char.character.json"
Exit status 7
ELIFECYCLE Command failed with exit code 7.

Also got this logs before the agent crashes:

◎ LOGS
[SttTtsPlugin] Plugin config =>
⛔ ERRORS
[Space] Error launching Space =>
{}
⛔ ERRORS
[Space] Error in routine =>
{}

y4my4my4m · 2025-01-21T23:07:41Z

@worksgoodcompany
FYI, ive been getting ERR_PNPM_RECURSIVE_RUN_FIRST_FAIL errors on the 0.1.9-alpha.1 which is why ive stayed on v0.1.8+build.1 for now.

tcm390 · 2025-01-22T00:00:24Z

Error: ICE connection failed
at RTCPeerConnection.

related to this? #1965

worksgoodcompany · 2025-01-22T05:31:39Z

Error: ICE connection failed
at RTCPeerConnection.

related to this? #1965

Yeah i checked that one too, but still didn't solve the issue. i'm using ubuntu 22 on wsl, and tried with all the suggestions on that issue, but still having problems. my guess is that it's related to agent-twitter-client and @roamhq/wrtc. Any suggestion on how to make twitter spaces work without issues?

Cooops · 2025-01-22T15:04:01Z

Im having this same issue myself, i had it working about 1.5-2 weeks ago off the main branch, which is the strangest part. Perhaps a minor config difference that is hard to capture via a generic error?

y4my4my4m · 2025-01-22T15:06:09Z

@Cooops my guess is that there's something that's trying to be printed in the "config" variable that doesn't exist.

Cooops · 2025-01-22T15:15:27Z

@y4my4my4m what the, yea just commented out that line and it worked -- back to talking with the space agent. so this log line causes issues when a certain config is missing, but we aren't sure which ones trigger it immediately based on the error alone?

edit: ah, i think its the transcription provider (at least in my case...there's some interesting logic that happens re: checking there, as i do have one configured). if thats the culprit for most, which i think it might be, i can work on a smol fix tonight?

worksgoodcompany · 2025-01-22T23:06:54Z

I tried with the latest develop branch version and now spaces starts, i can join, but when i try talking i get this logs:

[2025-01-22 23:04:36] ERROR: Max retry attempts reached:
totalAttempts: 3
error: "invalid input syntax for type uuid: "tw-1865462035586142208""
[2025-01-22 23:04:36] ERROR: Circuit breaker error in getRoomsForParticipants:
state: "CLOSED"
error: "invalid input syntax for type uuid: "tw-1865462035586142208""
[2025-01-22 23:04:36] ERROR (error): [SttTtsPlugin] processAudio error =>
length: 154
severity: "ERROR"
code: "22P02"
where: "unnamed portal parameter $1 = '...'"
file: "uuid.c"
line: "133"
routine: "string_to_uuid"

worksgoodcompany · 2025-01-22T23:59:57Z

I was able to fix the previous ""invalid input syntax for type uuid:" error by making changes and improving handling of user ids and memory storage. Now i was able to make the spaces work, or "kinda" work, because is still not consistent. It seems the ICE connection error occurs when the agent creates a spaces for the first time and no one joins. If i join the spaces, this error doesn't happen and the agent doesn't crash. I'll keep on debuggin

tcm390 · 2025-01-23T00:18:13Z

I was able to fix the previous ""invalid input syntax for type uuid:" error by making changes and improving handling of user ids and memory storage. Now i was able to make the spaces work, or "kinda" work, because is still not consistent. It seems the ICE connection error occurs when the agent creates a spaces for the first time and no one joins. If i join the spaces, this error doesn't happen and the agent doesn't crash. I'll keep on debuggin

what did you change for improving handling of user ids and memory storage? Could you make a pr?

worksgoodcompany · 2025-01-23T00:55:04Z

I was able to fix the previous ""invalid input syntax for type uuid:" error by making changes and improving handling of user ids and memory storage. Now i was able to make the spaces work, or "kinda" work, because is still not consistent. It seems the ICE connection error occurs when the agent creates a spaces for the first time and no one joins. If i join the spaces, this error doesn't happen and the agent doesn't crash. I'll keep on debuggin

what did you change for improving handling of user ids and memory storage? Could you make a pr?

Here is the PR with the changes i made #2686

tcm390 · 2025-01-23T01:05:58Z

I was able to fix the previous ""invalid input syntax for type uuid:" error by making changes and improving handling of user ids and memory storage. Now i was able to make the spaces work, or "kinda" work, because is still not consistent. It seems the ICE connection error occurs when the agent creates a spaces for the first time and no one joins. If i join the spaces, this error doesn't happen and the agent doesn't crash. I'll keep on debuggin

what did you change for improving handling of user ids and memory storage? Could you make a pr?

Here is the PR with the changes i made #2686

Reviewed, thanks

y4my4my4m added the bug Something isn't working label Jan 12, 2025

tcm390 self-assigned this Jan 14, 2025

AIFlowML added Needs Testing Need Feedback labels Jan 16, 2025

worksgoodcompany mentioned this issue Jan 20, 2025

feat(x spaces): Don't wait for mute, wait for silence #2576

Merged

worksgoodcompany mentioned this issue Jan 23, 2025

feat: [Space] improving handling of user ids and memory storage #2686

Merged

tcm390 mentioned this issue Jan 23, 2025

Remove the log as it may potentially cause the app to crash #2716

Merged

shakkernerd closed this as completed Jan 23, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SttTtsPlugin error when launching space #2206

SttTtsPlugin error when launching space #2206

y4my4my4m commented Jan 12, 2025

y4my4my4m commented Jan 12, 2025

daniellinuk commented Jan 14, 2025 •

edited

Loading

y4my4my4m commented Jan 14, 2025

daniellinuk commented Jan 15, 2025

CactusByte commented Jan 16, 2025 •

edited

Loading

y4my4my4m commented Jan 18, 2025

tcm390 commented Jan 18, 2025

daniellinuk commented Jan 18, 2025

y4my4my4m commented Jan 19, 2025

tcm390 commented Jan 19, 2025

tcm390 commented Jan 20, 2025

y4my4my4m commented Jan 21, 2025

worksgoodcompany commented Jan 21, 2025

worksgoodcompany commented Jan 21, 2025 •

edited

Loading

y4my4my4m commented Jan 21, 2025

tcm390 commented Jan 22, 2025

worksgoodcompany commented Jan 22, 2025

Cooops commented Jan 22, 2025 •

edited

Loading

y4my4my4m commented Jan 22, 2025

Cooops commented Jan 22, 2025 •

edited

Loading

worksgoodcompany commented Jan 22, 2025

worksgoodcompany commented Jan 22, 2025

tcm390 commented Jan 23, 2025

worksgoodcompany commented Jan 23, 2025

tcm390 commented Jan 23, 2025

SttTtsPlugin error when launching space #2206

SttTtsPlugin error when launching space #2206

Comments

y4my4my4m commented Jan 12, 2025

y4my4my4m commented Jan 12, 2025

daniellinuk commented Jan 14, 2025 • edited Loading

y4my4my4m commented Jan 14, 2025

daniellinuk commented Jan 15, 2025

CactusByte commented Jan 16, 2025 • edited Loading

y4my4my4m commented Jan 18, 2025

tcm390 commented Jan 18, 2025

daniellinuk commented Jan 18, 2025

y4my4my4m commented Jan 19, 2025

tcm390 commented Jan 19, 2025

tcm390 commented Jan 20, 2025

y4my4my4m commented Jan 21, 2025

worksgoodcompany commented Jan 21, 2025

worksgoodcompany commented Jan 21, 2025 • edited Loading

y4my4my4m commented Jan 21, 2025

tcm390 commented Jan 22, 2025

worksgoodcompany commented Jan 22, 2025

Cooops commented Jan 22, 2025 • edited Loading

y4my4my4m commented Jan 22, 2025

Cooops commented Jan 22, 2025 • edited Loading

worksgoodcompany commented Jan 22, 2025

worksgoodcompany commented Jan 22, 2025

tcm390 commented Jan 23, 2025

worksgoodcompany commented Jan 23, 2025

tcm390 commented Jan 23, 2025

daniellinuk commented Jan 14, 2025 •

edited

Loading

CactusByte commented Jan 16, 2025 •

edited

Loading

worksgoodcompany commented Jan 21, 2025 •

edited

Loading

Cooops commented Jan 22, 2025 •

edited

Loading

Cooops commented Jan 22, 2025 •

edited

Loading