Stuck in "Completing initialization…" when running from home directory

[e-gineer]

Describe the bug

My steampipe seems to get stuck here:

> select
  symbol,
  short_name,
  regular_market_price,
  regular_market_change_percent,
  regular_market_time
from
  finance_quote
where
  symbol in ('GME', 'BTC-USD', 'DOGE-USD', 'ETH-USD');
⠋ Completing initialization…

Steampipe version (steampipe -v)

v0.20.2

(v0.19.5 gets stuck as "Refreshing connections", but seems to be the same)

To reproduce

I could reproduce every time ... but then I switched directory away from home ... and it loaded ... and now I cannot reproduce again. I have no idea why.

*Notes

Trace logs from when it was failing:
database-2023-05-29.log
plugin-2023-05-29.log

Stray processes I found later ... I feel confident the CSV plugin is the source of issues:

~ $ ps aux | grep steampipe
nathan           44370   0.0  0.0 408486608    640 s003  R+    6:45AM   0:00.00 grep steampipe
nathan           43772   0.0  0.1 409315968  32432 s003  S     6:43AM   0:09.95 /Users/nathan/.steampipe/plugins/hub.steampipe.io/plugins/turbot/csv@latest/steampipe-plugin-csv.plugin
nathan           43044   0.0  0.1 409308080  31728 s003  S     6:41AM   0:09.38 /Users/nathan/.steampipe/plugins/hub.steampipe.io/plugins/turbot/csv@latest/steampipe-plugin-csv.plugin
nathan           42927   0.0  0.1 409315968  31568 s003  S     6:40AM   0:09.31 /Users/nathan/.steampipe/plugins/hub.steampipe.io/plugins/turbot/csv@latest/steampipe-plugin-csv.plugin
nathan           42808   0.0  0.1 409307264  32864 s003  S     6:40AM   0:09.31 /Users/nathan/.steampipe/plugins/hub.steampipe.io/plugins/turbot/csv@latest/steampipe-plugin-csv.plugin
nathan           42281   0.0  0.1 409308800  32224 s003  S     6:39AM   0:09.43 /Users/nathan/.steampipe/plugins/hub.steampipe.io/plugins/turbot/csv@latest/steampipe-plugin-csv.plugin
nathan           38959   0.0  0.1 409317248  26944 s003  S     6:31AM   0:10.71 /Users/nathan/.steampipe/plugins/hub.steampipe.io/plugins/turbot/csv@latest/steampipe-plugin-csv.plugin
nathan           37988   0.0  0.1 409316736  24848 s003  S     6:29AM   0:10.83 /Users/nathan/.steampipe/plugins/hub.steampipe.io/plugins/turbot/csv@latest/steampipe-plugin-csv.plugin
nathan           37590   0.0  0.1 409316992  27008 s003  S     6:28AM   0:09.86 /Users/nathan/.steampipe/plugins/hub.steampipe.io/plugins/turbot/csv@latest/steampipe-plugin-csv.plugin

Here is my CSV config, which should not be doing anything recursive (but feels like it is):

~ $ cat ~/.steampipe/config/csv.spc 
connection "csv" {
  plugin = "csv"
  paths = [ "*.csv" ]
}

Final observation ... the first time I saw this, my Mac prompted me twice to allow Terminal program access to sub-directories of the home directory - specifically Dropbox and Google Drive. I have no idea why it would be trying to access them? But that also makes me feel like it was looking recursively? I don't have any mod files in my home directory:

~ $ ls -al *.sp
ls: *.sp: No such file or directory

[binaek]

Underlying cause could be turbot/steampipe-plugin-sdk#557 due to which the CSV plugin is taking a long time to startup when running from the home directory, since it's trying to recurse down the dir tree of ~

[jdevoo]

I am having a similar behaviour with 0.20.5 - removing the csv dependency doesn't change anything for me.
I have to remove kubernetes, jira and csv and leave aws to have a regular session without that message when typing .inspect or .help...

below the log after a fresh start from within a container

2023-06-06 05:56:05.582 UTC [TRACE] steampipe: parse complete after 1 decode passes
2023-06-06 05:56:05.582 UTC [TRACE] steampipe: ensureInstallDir /home/steampipe/.steampipe
2023-06-06 05:56:05.812 UTC [TRACE] steampipe: Connection.PopulateChildren for aggregator connection aws_all
2023-06-06 05:56:05.812 UTC [TRACE] steampipe: Connection.PopulateChildren no connection matches aws_* - treating as a wildcard
2023-06-06 05:56:05.812 UTC [TRACE] steampipe: connection 'aws_default' matches pattern 'aws_*'
2023-06-06 05:56:05.812 UTC [TRACE] steampipe: connection 'aws_snapper' matches pattern 'aws_*'
2023-06-06 05:56:05.812 UTC [TRACE] steampipe: connection 'aws_minion' matches pattern 'aws_*'
2023-06-06 05:56:05.830 UTC [TRACE] steampipe: No memory limit set
Welcome to Steampipe v0.20.5
For more information, type .help
> 2023-06-06 05:56:05.844 UTC [TRACE] steampipe: no modfile exists in workspace folder - NOT creating pseudoresources and only loading resource files from top level folder
2023-06-06 05:56:05.996 UTC [TRACE] steampipe: parse complete after 1 decode passes
2023-06-06 05:56:06.147 UTC [TRACE] steampipe: parse complete after 1 decode passes
2023-06-06 05:56:06.147 UTC [TRACE] steampipe: Workspace load took 316ms
2023-06-06 05:56:06.147 UTC [TRACE] steampipe: telemetry.Init service 'steampipe', tracingEnabled: false, metricsEnabled: false
2023-06-06 05:56:06.147 UTC [TRACE] steampipe: metrics and tracing disabled' - returning
2023-06-06 05:56:06.155 UTC [TRACE] steampipe: GetRunStatus - loadRunningInstanceInfo returned nil
2023-06-06 05:56:06.155 UTC [TRACE] steampipe: StartDB invoker query
2023-06-06 05:56:06.158 UTC [TRACE] steampipe: Trying to create client with:  host=localhost port=9193 user=root dbname=postgres sslmode=disable
2023-06-06 05:56:06.159 UTC [TRACE] steampipe: could not connect: failed to connect to `host=localhost user=root database=postgres`: dial error (dial tcp [::1]:9193: connect: cannot assign requested address)
2023-06-06 05:56:06.244 UTC [TRACE] steampipe: SERVICE: 2023-06-06 05:56:06.243 UTC [19] LOG:  redirecting log output to logging collector process
2023-06-06 05:56:06.247 UTC [TRACE] steampipe: SERVICE: 2023-06-06 05:56:06.243 UTC [19] HINT:  Future log output will appear in directory "/home/steampipe/.steampipe/logs".
2023-06-06 05:56:06.360 UTC [TRACE] steampipe: Trying to create client with:  host=localhost port=9193 user=root dbname=postgres sslmode=disable
2023-06-06 05:56:06.432 UTC [TRACE] steampipe: connected to database
2023-06-06 05:56:06.432 UTC [TRACE] steampipe: Pinging
2023-06-06 05:56:06.432 UTC [TRACE] steampipe: checking for recovery mode
2023-06-06 05:56:06.519 UTC [TRACE] steampipe: SQLInfoMap >>> map[dbname:postgres host:localhost port:9193 sslcert:/home/steampipe/.steampipe/db/14.2.0/data/server.crt sslkey:/home/steampipe/.steampipe/db/14.2.0/data/server.key sslmode:verify-ca sslrootcert:/home/steampipe/.steampipe/db/14.2.0/data/root.crt user:root]
2023-06-06 05:56:06.520 UTC [TRACE] steampipe: PSQLInfo >>> [sslrootcert=/home/steampipe/.steampipe/db/14.2.0/data/root.crt sslcert=/home/steampipe/.steampipe/db/14.2.0/data/server.crt user=root dbname=postgres host=localhost port=9193 sslkey=/home/steampipe/.steampipe/db/14.2.0/data/server.key sslmode=verify-ca]
2023-06-06 05:56:06.535 UTC [TRACE] steampipe: Pinging
2023-06-06 05:56:06.608 UTC [TRACE] steampipe: SQLInfoMap >>> map[dbname:steampipe host:localhost port:9193 sslcert:/home/steampipe/.steampipe/db/14.2.0/data/server.crt sslkey:/home/steampipe/.steampipe/db/14.2.0/data/server.key sslmode:verify-ca sslrootcert:/home/steampipe/.steampipe/db/14.2.0/data/root.crt user:root]
2023-06-06 05:56:06.608 UTC [TRACE] steampipe: PSQLInfo >>> [sslcert=/home/steampipe/.steampipe/db/14.2.0/data/server.crt user=root dbname=steampipe host=localhost port=9193 sslkey=/home/steampipe/.steampipe/db/14.2.0/data/server.key sslmode=verify-ca sslrootcert=/home/steampipe/.steampipe/db/14.2.0/data/root.crt]
2023-06-06 05:56:06.702 UTC [TRACE] steampipe: Pinging
2023-06-06 05:56:07.093 UTC [TRACE] steampipe: start: started plugin manager, pid 31
2023-06-06 05:56:07.094 UTC [TRACE] steampipe: SQLInfoMap >>> map[dbname:steampipe host:localhost port:9193 sslcert:/home/steampipe/.steampipe/db/14.2.0/data/server.crt sslkey:/home/steampipe/.steampipe/db/14.2.0/data/server.key sslmode:verify-ca sslrootcert:/home/steampipe/.steampipe/db/14.2.0/data/root.crt user:root]
2023-06-06 05:56:07.094 UTC [TRACE] steampipe: PSQLInfo >>> [dbname=steampipe host=localhost sslmode=verify-ca sslrootcert=/home/steampipe/.steampipe/db/14.2.0/data/root.crt sslcert=/home/steampipe/.steampipe/db/14.2.0/data/server.crt sslkey=/home/steampipe/.steampipe/db/14.2.0/data/server.key port=9193 user=root]
2023-06-06 05:56:07.103 UTC [TRACE] steampipe: Pinging
2023-06-06 05:56:07.104 UTC [TRACE] steampipe: querying for legacy 'internal' schema
2023-06-06 05:56:07.220 UTC [TRACE] steampipe: could not find any objects in 'internal' - skipping drop
2023-06-06 05:56:07.220 UTC [TRACE] steampipe: could not find legacy 'internal' schema
2023-06-06 05:56:07.424 UTC [TRACE] steampipe: SQLInfoMap >>> map[dbname:steampipe host:localhost port:9193 sslcert:/home/steampipe/.steampipe/db/14.2.0/data/server.crt sslkey:/home/steampipe/.steampipe/db/14.2.0/data/server.key sslmode:verify-ca sslrootcert:/home/steampipe/.steampipe/db/14.2.0/data/root.crt user:root]
2023-06-06 05:56:07.424 UTC [TRACE] steampipe: PSQLInfo >>> [host=localhost port=9193 user=root dbname=steampipe sslmode=verify-ca sslrootcert=/home/steampipe/.steampipe/db/14.2.0/data/root.crt sslcert=/home/steampipe/.steampipe/db/14.2.0/data/server.crt sslkey=/home/steampipe/.steampipe/db/14.2.0/data/server.key]
2023-06-06 05:56:07.434 UTC [TRACE] steampipe: Pinging
2023-06-06 05:56:07.441 UTC [TRACE] steampipe: SQLInfoMap >>> map[dbname:steampipe host:localhost port:9193 sslcert:/home/steampipe/.steampipe/db/14.2.0/data/server.crt sslkey:/home/steampipe/.steampipe/db/14.2.0/data/server.key sslmode:verify-ca sslrootcert:/home/steampipe/.steampipe/db/14.2.0/data/root.crt user:root]
2023-06-06 05:56:07.441 UTC [TRACE] steampipe: PSQLInfo >>> [sslrootcert=/home/steampipe/.steampipe/db/14.2.0/data/root.crt host=localhost port=9193 user=root dbname=steampipe sslcert=/home/steampipe/.steampipe/db/14.2.0/data/server.crt sslkey=/home/steampipe/.steampipe/db/14.2.0/data/server.key sslmode=verify-ca]
2023-06-06 05:56:07.454 UTC [TRACE] steampipe: Pinging
2023-06-06 05:56:07.460 UTC [TRACE] steampipe: plugin manager is running - returning client
2023-06-06 05:56:07.475 UTC [TRACE] steampipe: SQLInfoMap >>> map[dbname:steampipe host:localhost port:9193 sslcert:/home/steampipe/.steampipe/db/14.2.0/data/server.crt sslkey:/home/steampipe/.steampipe/db/14.2.0/data/server.key sslmode:verify-ca sslrootcert:/home/steampipe/.steampipe/db/14.2.0/data/root.crt user:steampipe]
2023-06-06 05:56:07.475 UTC [TRACE] steampipe: PSQLInfo >>> [sslcert=/home/steampipe/.steampipe/db/14.2.0/data/server.crt sslkey=/home/steampipe/.steampipe/db/14.2.0/data/server.key sslmode=verify-ca host=localhost port=9193 user=steampipe dbname=steampipe sslrootcert=/home/steampipe/.steampipe/db/14.2.0/data/root.crt]
2023-06-06 05:56:07.535 UTC [TRACE] steampipe: Pinging
2023-06-06 05:56:07.539 UTC [TRACE] steampipe: created local client 0xc0005f40a0
> 2023-06-06 05:57:07.903 UTC [TRACE] steampipe: SetupWatcher
2023-06-06 05:57:08.030 UTC [TRACE] steampipe: Start notifications listener
2023-06-06 05:57:08.030 UTC [TRACE] steampipe: initialiseSuggestions
2023-06-06 05:57:08.030 UTC [TRACE] steampipe: InteractiveClient listenToPgNotifications
2023-06-06 05:57:08.031 UTC [TRACE] steampipe: SQLInfoMap >>> map[dbname:steampipe host:localhost port:9193 sslcert:/home/steampipe/.steampipe/db/14.2.0/data/server.crt sslkey:/home/steampipe/.steampipe/db/14.2.0/data/server.key sslmode:verify-ca sslrootcert:/home/steampipe/.steampipe/db/14.2.0/data/root.crt user:steampipe]
2023-06-06 05:57:08.031 UTC [TRACE] steampipe: PSQLInfo >>> [dbname=steampipe sslrootcert=/home/steampipe/.steampipe/db/14.2.0/data/root.crt sslcert=/home/steampipe/.steampipe/db/14.2.0/data/server.crt sslkey=/home/steampipe/.steampipe/db/14.2.0/data/server.key sslmode=verify-ca host=localhost port=9193 user=steampipe]
2023-06-06 05:57:08.045 UTC [TRACE] steampipe: Pinging
2023-06-06 05:57:08.046 UTC [TRACE] steampipe: Wait for notification

then I issue .help

> .help
2023-06-06 05:59:34.282 UTC [TRACE] steampipe: failed to load connection state - are we connected to a server running a previous steampipe version?
Welcome to Steampipe shell.

To start, simply enter your SQL query at the prompt:

  select * from aws_iam_user

Common commands:

.help     Show steampipe help
.inspect  View connections, tables & column information
.exit     Exit from steampipe terminal

Advanced commands:

.autocomplete on|off Enable or disable auto-completion
.cache [mode]        Enable, disable or clear the query cache
.cache_ttl           Set the cache ttl (time-to-live)
.clear               Clear the console
.connections         List active connections
.header on|off       Enable or disable column headers
.multi on|off        Enable or disable multiline mode
.output [mode]       Set output format: csv, json, table or line
.quit                Exit from steampipe terminal
.search_path         Display the current search path, or set the search-path by passing in a comma-separated list
.search_path_prefix  Set a prefix to the current search-path
.separator           Set csv output separator
.tables              List or describe tables
.timing on|off       Enable or disable query execution timing

Documentation available at https://steampipe.io/docs

>

[jdevoo]

Looks like my problem came from a multi-account aws.spc which defined an aggregator using wildcard. After specifying each connection the container launches more quickly.

[kaidaguerre]

The original issue was fixed by SDK update v5.5.0 (turbot/steampipe-plugin-sdk@a990cfd)