Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Damselfly keeps crashing #528

Open
gouthamravee opened this issue Apr 26, 2024 · 12 comments
Open

Damselfly keeps crashing #528

gouthamravee opened this issue Apr 26, 2024 · 12 comments

Comments

@gouthamravee
Copy link

Damselfly version: v4.1.3.0

docker compose

services:
  damselfly:
    container_name: damselfly
    image: webreaper/damselfly
    network_mode: host
    environment:
      - TZ="America/New_York"
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ./config:/config
      - ./thumbs:/thumbs
      - /mnt/photography:/pictures
    restart: unless-stopped

I added my whole 1.8TB photography collection to damselfly and I keep seeing messages like what you see below. I have AI object detection and face recognition turned off. Thumbnail generation is running.

Damselfly is running in docker, inside a Debian based VM in proxmox. The VM has 20 cores and 15GB of ram assigned to it.

Do these errors just mean damselfly doesn't have the thumbnails for the images yet? Is that related to the 'killed' message, and what seems like a crash. Ram usage is very low, only about 9% of total ram is being actively used.

damselfly  | [03:04:09.670-.NET TP Worker-ERR] Unable to process /thumb/Large/47038: Object reference not set to an instance of an object.
damselfly  | [03:04:09.674-.NET TP Worker-ERR] Thumbnail conversion failed for /pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2024/2024-04-20/_DSC1910.dng: Object reference not set to an instance of an object.
damselfly  | [03:04:09.701-.NET TP Worker-ERR] Unable to process /thumb/Large/47490: Object reference not set to an instance of an object.
damselfly  | Killed
damselfly exited with code 137
damselfly  | Preparing to start Damselfly....
@Webreaper
Copy link
Owner

Are you able to attach the image it's processing when it fails? Or can you email it to me at [email protected]?

@Webreaper
Copy link
Owner

Webreaper commented Apr 26, 2024 •
edited
Loading

I've pushed a potential fix for this (entirely speculative as I can't repro). Can you give it a go by pulling the dev version (pull webreaper/damselfly:dev) once the build has completed (here: https://github.com/Webreaper/Damselfly/actions/runs/8843240288)?

@gouthamravee
Copy link
Author

gouthamravee commented Apr 26, 2024 •
edited
Loading

@Webreaper I'm not sure which image its failing at, To make sure I send you the right one, in the logs I posted would it be the last image it couldn't find thumbnails for?

damselfly  | [03:04:09.674-.NET TP Worker-ERR] Thumbnail conversion failed for /pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2024/2024-04-20/_DSC1910.dng: Object reference not set to an instance of an object.
damselfly  | [03:04:09.701-.NET TP Worker-ERR] Unable to process /thumb/Large/47490: Object reference not set to an instance of an object.

This is another error I've seen often, I believe this happens when ever I refresh the web page for damesfly when its going through my reverse proxy.

damselfly  | [14:14:16.507-.NET TP Worker-WRN] Failed to determine the https port for redirect.
damselfly  | [14:14:18.219-.NET TP Worker-ERR] An unhandled exception has occurred while executing the request.
damselfly  | System.InvalidOperationException: Unable to resolve service for type 'Microsoft.AspNetCore.ApiAuthorization.IdentityServer.IClientRequestParametersProvider' while attempting to activate 'Damselfly.Web.Server.Controllers.OidcConfigurationController'.
damselfly  |    at Microsoft.Extensions.DependencyInjection.ActivatorUtilities.ThrowHelperUnableToResolveService(Type type, Type requiredBy)
damselfly  |    at lambda_method1131(Closure, IServiceProvider, Object[])
damselfly  |    at Microsoft.AspNetCore.Mvc.Controllers.ControllerFactoryProvider.<>c__DisplayClass6_0.<CreateControllerFactory>g__CreateController|0(ControllerContext controllerContext)
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()
damselfly  | --- End of stack trace from previous location ---
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()
damselfly  | --- End of stack trace from previous location ---
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
damselfly  |    at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
damselfly  |    at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
damselfly  |    at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.InvokeCore(HttpContext context)
damselfly  |    at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)

I will try the dev version and let you know, there are still around 20k images left to process for thumbnails.

@gouthamravee
Copy link
Author

gouthamravee commented Apr 26, 2024 •
edited
Loading

With the dev version I'm not seeing a crash after about 15 minutes of it running. I am still seeing the errors about the thumbnails but the application doesn't seem to be getting killed. I'm also seeing thumbnails loading on the web page.

A bit more detail, I'm using a reverse proxy (nginx) to access damesfly through my public domain. It seems like something about that setup isn't working absolutely perfectly.

In addition, the path for all my photos to damselfly is a NFS share. The images aren't physically on the same hardware as damselfly. The images are on another server sitting right next to the server damselfly is running on. There's 4x 1Gbs network connections between the two systems.

Here's the log I'm seeing now.

damselfly  | Preparing to start Damselfly....
damselfly  |   ./Damselfly.Web.Server /pictures --config=/config --thumbs=/thumbs
damselfly  | [14:22:05.995-0001-INF] === Damselfly Blazor Server Log Started ===
damselfly  | [14:22:06.317-0001-INF] Running Sqlite DB migrations...
damselfly  | [14:22:09.148-0001-INF] Running Sqlite DB optimisation...
damselfly  | [14:22:10.650-0001-INF] DB optimisation complete.
damselfly  | [14:22:10.863-0001-INF] Full-text search index rebuilt.
damselfly  | [14:22:10.875-0001-INF] Preloading config cache with all settings...
damselfly  | [14:22:10.965-0001-INF] Loaded 27 settings into config cache.
damselfly  | [14:22:11.096-0001-INF] Preloading config cache with all settings...
damselfly  | [14:22:11.211-0001-INF] Loaded 27 settings into config cache.
damselfly  | [14:22:11.873-0001-INF]   Executing: convert --version
damselfly  | [14:22:11.909-0001-INF] ImageMagick found: vVersion: ImageMagick 7.1.1-29 Q16-HDRI x86_64 21991 https://imagemagick.org
damselfly  | [14:22:11.937-0001-INF] Watermark font installed: /app/wwwroot/fonts/arial.ttf
damselfly  | [14:22:11.940-0001-INF] Created downloads folder: /app/wwwroot/downloads
damselfly  | [14:22:11.944-0001-INF] Task scheduler started.
damselfly  | [14:22:11.950-0001-INF] Registered job processing source: ThumbnailService
damselfly  | [14:22:11.952-0001-INF]   Executing: exiftool -ver
damselfly  | [14:22:12.139-0001-INF] ExifTool Version: v12.78
damselfly  |
damselfly  | [14:22:12.210-0001-INF] Registered job processing source: ExifService
damselfly  | [14:22:12.212-0001-INF] Warming up image cache with up to 2000 most recent images.
damselfly  | [14:22:15.503-0001-INF] Enriched and cached 2000 in 3238ms
damselfly  | [14:22:15.504-0001-INF] Image Cache primed with 2000 images.
damselfly  | [14:22:15.504-0001-INF] Started Work service thread.
damselfly  | [14:22:15.543-0001-INF] Registered job processing source: MetaDataService
damselfly  | [14:22:15.544-0001-INF] Registered job processing source: IndexingService
damselfly  | [14:22:17.733-0001-INF] Registered job processing source: ImageRecognitionService
damselfly  | [14:22:18.917-0001-INF] Adding scheduled task: CleanupThumbs every 1 week
damselfly  | [14:22:18.923-0001-INF] Adding scheduled task: CleanupDownloads every 6 hours
damselfly  | [14:22:18.923-0001-INF] Adding scheduled task: CleanupKeywordOps every 12 hours
damselfly  | [14:22:18.923-0001-INF] Adding scheduled task: DumpPerformance every 1 day
damselfly  | [14:22:18.924-0001-INF] Startup complete. Reducing console logging level to [Warning/Error].
damselfly  | [14:22:18.963-0001-WRN] No XML encryptor configured. Key {f16ca4a0-534b-43eb-bf85-b30109a47de0} may be persisted to storage in unencry
damselfly  | [14:22:48.180-.NET TP Worker-WRN] Failed to determine the https port for redirect.
damselfly  | [14:36:20.852-.NET TP Worker-ERR] An unhandled exception has occurred while executing the request.
damselfly  | System.InvalidOperationException: Unable to resolve service for type 'Microsoft.AspNetCore.ApiAuthorization.IdentityServer.IClientRequestParametersProvider' while attempting to activate 'Damselfly.Web.Server.Controllers.OidcConfigurationController'.
damselfly  |    at Microsoft.Extensions.DependencyInjection.ActivatorUtilities.ThrowHelperUnableToResolveService(Type type, Type requiredBy)
damselfly  |    at lambda_method1141(Closure, IServiceProvider, Object[])
damselfly  |    at Microsoft.AspNetCore.Mvc.Controllers.ControllerFactoryProvider.<>c__DisplayClass6_0.<CreateControllerFactory>g__CreateController|0(ControllerContext controllerContext)
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()
damselfly  | --- End of stack trace from previous location ---
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()
damselfly  | --- End of stack trace from previous location ---
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
damselfly  |    at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
damselfly  |    at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
damselfly  |    at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.InvokeCore(HttpContext context)
damselfly  |    at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
damselfly  | [14:36:32.515-.NET TP Worker-ERR] An unhandled exception has occurred while executing the request.
damselfly  | System.InvalidOperationException: Unable to resolve service for type 'Microsoft.AspNetCore.ApiAuthorization.IdentityServer.IClientRequestParametersProvider' while attempting to activate 'Damselfly.Web.Server.Controllers.OidcConfigurationController'.
damselfly  |    at Microsoft.Extensions.DependencyInjection.ActivatorUtilities.ThrowHelperUnableToResolveService(Type type, Type requiredBy)
damselfly  |    at lambda_method1141(Closure, IServiceProvider, Object[])
damselfly  |    at Microsoft.AspNetCore.Mvc.Controllers.ControllerFactoryProvider.<>c__DisplayClass6_0.<CreateControllerFactory>g__CreateController|0(ControllerContext controllerContext)
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()
damselfly  | --- End of stack trace from previous location ---
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()
damselfly  | --- End of stack trace from previous location ---
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
damselfly  |    at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
damselfly  |    at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
damselfly  |    at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.InvokeCore(HttpContext context)
damselfly  |    at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
damselfly  | [14:36:53.552-.NET TP Worker-ERR] Thumbnail conversion failed for /pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2024/2024-04-20/_DSC1937-HDR.dng: Object reference not set to an instance of an object.
damselfly  | [14:36:53.580-.NET TP Worker-ERR] Thumbnail conversion failed for /pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2024/2024-04-20/_DSC1920-HDR.dng: Object reference not set to an instance of an object.
damselfly  | [14:36:53.590-.NET TP Worker-ERR] Thumbnail conversion failed for /pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2024/2024-04-21/_DSC1957-HDR.dng: Object reference not set to an instance of an object.
damselfly  | [14:36:53.593-.NET TP Worker-ERR] Thumbnail conversion failed for /pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2024/2024-04-20/_DSC1926-HDR.dng: Object reference not set to an instance of an object.
damselfly  | [14:36:53.594-.NET TP Worker-ERR] Thumbnail conversion failed for /pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2024/2024-04-20/_DSC1904-HDR.dng: Object reference not set to an instance of an object.
damselfly  | [14:36:53.636-.NET TP Worker-ERR] Thumbnail conversion failed for /pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2024/2024-04-20/_DSC1914-HDR.dng: Object reference not set to an instance of an object.
damselfly  | [14:36:53.946-.NET TP Worker-ERR] Thumbnail conversion failed for /pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2024/2024-04-21/_DSC2181.dng: Object reference not set to an instance of an object.
damselfly  | [14:37:00.083-.NET TP Worker-ERR] Thumbnail conversion failed for /pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2024/2024-04-21/_DSC2179.dng: Object reference not set to an instance of an object.
damselfly  | [14:37:00.099-.NET TP Worker-ERR] Thumbnail conversion failed for /pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2024/2024-04-20/_DSC1897-HDR.dng: Object reference not set to an instance of an object.
damselfly  | [14:39:05.172-.NET TP Worker-WRN] HTTP Exception: TypeError: NetworkError when attempting to fetch resource. (/api/config)
damselfly  | [14:39:05.173-.NET TP Worker-WRN]   Inner Exception: TypeError: NetworkError when attempting to fetch resource.
damselfly  | [14:39:05.174-.NET TP Worker-WRN] HTTP Exception: TypeError: NetworkError when attempting to fetch resource. (/api/config)
damselfly  | [14:39:05.174-.NET TP Worker-WRN]   Inner Exception: TypeError: NetworkError when attempting to fetch resource.
damselfly  | [14:38:25.708-.NET TP Worker-ERR] An unhandled exception has occurred while executing the request.
damselfly  | System.InvalidOperationException: Unable to resolve service for type 'Microsoft.AspNetCore.ApiAuthorization.IdentityServer.IClientRequestParametersProvider' while attempting to activate 'Damselfly.Web.Server.Controllers.OidcConfigurationController'.
damselfly  |    at Microsoft.Extensions.DependencyInjection.ActivatorUtilities.ThrowHelperUnableToResolveService(Type type, Type requiredBy)
damselfly  |    at lambda_method1141(Closure, IServiceProvider, Object[])
damselfly  |    at Microsoft.AspNetCore.Mvc.Controllers.ControllerFactoryProvider.<>c__DisplayClass6_0.<CreateControllerFactory>g__CreateController|0(ControllerContext controllerContext)
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()
damselfly  | --- End of stack trace from previous location ---
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()
damselfly  | --- End of stack trace from previous location ---
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
damselfly  |    at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
damselfly  |    at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
damselfly  |    at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
damselfly  |    at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.InvokeCore(HttpContext context)
damselfly  |    at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
damselfly  | [14:38:36.299-.NET TP Worker-ERR] Thumbnail conversion failed for /pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2024/2024-04-20/_DSC1937-HDR.dng: Object reference not set to an instance of an object.
damselfly  | [14:38:36.301-.NET TP Worker-ERR] Thumbnail conversion failed for /pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2024/2024-04-20/_DSC1926-HDR.dng: Object reference not set to an instance of an object.
damselfly  | [14:38:36.425-.NET TP Worker-ERR] Thumbnail conversion failed for /pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2024/2024-04-20/_DSC1920-HDR.dng: Object reference not set to an instance of an object.
damselfly  | [14:38:37.114-.NET TP Worker-ERR] Thumbnail conversion failed for /pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2024/2024-04-20/_DSC1914-HDR.dng: Object reference not set to an instance of an object.
damselfly  | [14:38:37.241-.NET TP Worker-ERR] Thumbnail conversion failed for /pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2024/2024-04-21/_DSC1957-HDR.dng: Object reference not set to an instance of an object.
damselfly  | [14:38:37.243-.NET TP Worker-ERR] Thumbnail conversion failed for /pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2024/2024-04-20/_DSC1904-HDR.dng: Object reference not set to an instance of an object.
damselfly  | [14:38:37.525-.NET TP Worker-ERR] Thumbnail conversion failed for /pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2024/2024-04-21/_DSC2179.dng: Object reference not set to an instance of an object.
damselfly  | [14:38:37.887-.NET TP Worker-ERR] Thumbnail conversion failed for /pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2024/2024-04-20/_DSC1897-HDR.dng: Object reference not set to an instance of an object.
damselfly  | [14:38:38.799-.NET TP Worker-ERR] Thumbnail conversion failed for /pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2024/2024-04-21/_DSC2181.dng: Object reference not set to an instance of an object.

The HTTP and System.InvalidOperationException: Unable to resolve service for type errors happen when ever I refresh the web page for damselfly.

Weird thing now is even though thumbnails are loading, if I try to open the image by double clicking I get an "image not found" error.

Edit:

Updated too soon, it did end up crashing again.

damselfly  | [14:38:37.887-.NET TP Worker-ERR] Thumbnail conversion failed for /pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2024/2024-04-20/_DSC1897-HDR.dng: Object reference not set to an instance of an object.
damselfly  | [14:38:38.799-.NET TP Worker-ERR] Thumbnail conversion failed for /pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2024/2024-04-21/_DSC2181.dng: Object reference not set to an instance of an object.
damselfly  | Killed
damselfly exited with code 137
damselfly  | Preparing to start Damselfly....
damselfly  |   ./Damselfly.Web.Server /pictures --config=/config --thumbs=/thumbs
damselfly  | [14:49:14.046-0001-INF] === Damselfly Blazor Server Log Started ===
damselfly  | [14:49:14.434-0001-INF] Running Sqlite DB migrations...
damselfly  | [14:49:17.049-0001-INF] Running Sqlite DB optimisation...

Another update, sorry for all these edits.

This time it looks like some resource usage caused the whole VM to lock up, when I started damsefly up again I saw a new error.

damselfly  | [15:36:44.108-.NET TP Worker-ERR] Thumbnail conversion failed for /pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2022/2022-12-22/_DSC6513.dng: Object reference not set to an instance of an object.
damselfly  | [15:36:44.337-.NET TP Worker-ERR] Thumbnail conversion failed for /pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2022/2022-12-22/_DSC6523.dng: Object reference not set to an instance of an object.
damselfly  | [15:42:08.982-Kestrel Timer-WRN] As of "04/26/2024 15:38:57 +00:00", the heartbeat has been running for "00:00:02.1575431" which is longer than "00:00:01". This could be caused by thread pool starvation.
damselfly  | [15:42:13.369-.NET TP Worker-ERR] Conversion failed. Unable to start process: Failed
damselfly  | [15:42:13.378-.NET TP Worker-ERR] Failed commandline was: convert  -define jpeg:size=800x800 "/pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2022/2022-03-12/_DSC9872-Pano-Edit.tif" -quality 90 -unsharp 0.5x0.5+1.25+0.0 -thumbnail 800x800 -auto-orient -write "/thumbs/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2022/2022-03-12/_DSC9872-Pano-Edit_l.JPG"  -thumbnail 320x320 -auto-orient -write "/thumbs/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2022/2022-03-12/_DSC9872-Pano-Edit_m.JPG"   -extent 120:120 -gravity center -thumbnail 120x120 -auto-orient "/thumbs/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2022/2022-03-12/_DSC9872-Pano-Edit_s.JPG"

@Webreaper
Copy link
Owner

Not sure how much I can help with the proxy thing. Never had that reported, and I won't be able to repro.

For the crash, can you send me _DSC1897-HDR.dng, _DSC2181.dng, _DSC6523.dng and _DSC6523.dng so I can test why they're causing the crash? (or just send one, and then I can see if i can repro).

@gouthamravee
Copy link
Author

@Webreaper

No worries on the reverse proxy thing, I can troubleshoot that on my own.

I've emailed you the requested pictures, its only 3 so I don't mind sending you more data to test with.

@gouthamravee
Copy link
Author

gouthamravee commented Apr 26, 2024 •
edited
Loading

After a few hours it seems like a memory leak crashed most of the VM. The last logs in netdata show a spike in memory usage, and proxmox shows a ton of disk writes before the crash.

The last logs are

17:49:59.710 	WRN 	As of ""2024-04-26T17:49:52.0714653+00:00"", the heartbeat has been running for ""00:00:01.1135167"" which is longer than ""00:00:01"". This could be caused by thread pool starvation.
17:48:50.015 	ERR 	System.Net.Http.HttpRequestException: net_http_message_not_success_statuscode_reason, 504, Gateway Timeout
17:48:50.011 	WRN 	HTTP Exception: net_http_message_not_success_statuscode_reason, 504, Gateway Timeout (/api/data/stats)
17:48:41.145 	ERR 	System.Net.Http.HttpRequestException: net_http_message_not_success_statuscode_reason, 504, Gateway Timeout
17:48:41.142 	WRN 	HTTP Exception: net_http_message_not_success_statuscode_reason, 504, Gateway Timeout (/api/logs/0)
17:48:36.496 	ERR 	System.Net.Http.HttpRequestException: net_http_message_not_success_statuscode_reason, 504, Gateway Timeout
17:48:36.488 	WRN 	HTTP Exception: net_http_message_not_success_statuscode_reason, 504, Gateway Timeout (/api/data/newversion)
17:48:35.325 	WRN 	As of ""2024-04-26T17:46:32.6490334+00:00"", the heartbeat has been running for ""00:00:01.5247904"" which is longer than ""00:00:01"". This could be caused by thread pool starvation.
17:11:42.641 	ERR 	Failed commandline was: convert -define jpeg:size=800x800 "/pictures/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2021/2021-04-03/_DSC8841.ARW" -quality 90 -unsharp 0.5x0.5+1.25+0.0 -thumbnail 800x800 -auto-orient -write "/thumbs/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2021/2021-04-03/_DSC8841_l.JPG" -thumbnail 320x320 -auto-orient -write "/thumbs/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2021/2021-04-03/_DSC8841_m.JPG" -extent 120:120 -gravity center -thumbnail 120x120 -auto-orient "/thumbs/Lightroom CC/3da420dce54846ac9a5ef36da157ce47/originals/2021/2021-04-03/_DSC8841_s.JPG"
17:11:42.641 	ERR 	Conversion failed. Unable to start process: Failed
17:11:15.317 	INF 	Full index compelete.
17:02:09.474 	INF 	Performing full index.

Tried this again with 24GB of ram to the VM and same thing happened.

@gouthamravee
Copy link
Author

@Webreaper any luck in recreating this?

@Webreaper
Copy link
Owner

Sorry, stuff cropped up so haven't had time. Still on my list to investigate.

@gouthamravee
Copy link
Author

No worries, let me know if you need anything else when you get to it. Thank you.

@gouthamravee
Copy link
Author

gouthamravee commented Jun 4, 2024 •
edited
Loading

@Webreaper Throwing more RAM at it seems to have solved the issue. I gave the VM 24GB and added another 64GB to the host so there's more room for all the VMs running there. Damselfly seems to have finished all the indexing tasks for the 7000 new images I've added since the issue started.

I'm letting it do the AI processing now.

@Webreaper
Copy link
Owner

It's a bit hungry. 😁

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Webreaper @gouthamravee