Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Create new task not working after CVAT upgrade #7377

Closed
2 tasks done
kgramzinski opened this issue Jan 19, 2024 · 2 comments
Closed
2 tasks done

Create new task not working after CVAT upgrade #7377

kgramzinski opened this issue Jan 19, 2024 · 2 comments
Labels
bug Something isn't working need info Need more information to investigate the issue

Comments

@kgramzinski
Copy link

Actions before raising this issue

  • I searched the existing issues and did not find anything similar.
  • I read/searched the docs

Steps to Reproduce

FROM UI

  1. Create new task
  2. Drag and drop local files (images or video)
  3. Submit task
  4. Images are uploading
  5. Failure - File not found error: no such file or directory: /home/django/data/data/1070/raw/12_06_WV03_VNIR_R1C1_tile_11_25.jpg

After this, inside the container, I can see that the folder /home/django/data/data/1070 does NOT exist.

FROM COMMAND LINE API IN PYTHON

  1. task_id = create_new_task(auth, cvat_taskname, user_id, project_id, image_quality)

  2. data = {
    "image_quality": image_quality,
    "compressed_chunk_type": "imageset",
    "original_chunk_type": "imageset"
    }

    image_paths = []
    for image in os.listdir(image_folder):
    image_paths.append(os.path.join(image_folder, image))

    files = {f'client_files[{i}]': open(f, 'rb') for i, f in enumerate(image_paths)}

    req = server + api + tasks + f'/{task_id}/data'
    upload_response = requests.post(req, data=data, files=files, auth=auth)

  3. Failure
    File "/home/django/cvat/apps/engine/media_extractors.py", line 215, in get_image_size
    with Image.open(self._source_path[i]) as img:
    File "/opt/venv/lib/python3.10/site-packages/PIL/Image.py", line 3243, in open
    fp = builtins.open(filename, "rb")
    FileNotFoundError: [Errno 2] No such file or directory: '/home/django/data/data/1076/raw/12_06_WV03_VNIR_R1C1_tile_11_25

After this, inside the container, I can see that the folder /home/django/data/data/1070 DOES exist. I also tried re-uploading data to the same task using the API in Python:

  1. task_id = same as above

  2. data = {
    "image_quality": image_quality,
    "compressed_chunk_type": "imageset",
    "original_chunk_type": "imageset"
    }

    image_paths = []
    for image in os.listdir(image_folder):
    image_paths.append(os.path.join(image_folder, image))

    files = {f'client_files[{i}]': open(f, 'rb') for i, f in enumerate(image_paths)}

    req = server + api + tasks + f'/{task_id}/data'
    upload_response = requests.post(req, data=data, files=files, auth=auth)

  3. Failure - response 500
    File "/home/django/cvat/apps/engine/media_extractors.py", line 215, in get_image_size
    with Image.open(self._source_path[i]) as img:
    File "/opt/venv/lib/python3.10/site-packages/PIL/Image.py", line 3243, in open
    fp = builtins.open(filename, "rb")
    FileNotFoundError: [Errno 2] No such file or directory: '/home/django/data/data/1076/raw/12_06_WV03_VNIR_R1C1_tile_11_25.jpg'

However, I can see that this file does in fact exist in the cvat_server container.
image

Before this, I had upgraded my version of CVAT following the instructions here:
https://opencv.github.io/cvat/docs/administration/advanced/upgrade_guide/
https://opencv.github.io/cvat/docs/administration/advanced/upgrade_guide/#how-to-upgrade-postgresql-database-base-image
https://opencv.github.io/cvat/docs/administration/basics/installation/#how-to-pullbuildupdate-cvat-images

I can see all of my old data in CVAT and open up old tasks/jobs, I just can't create new ones.

Expected Behavior

After upgrading CVAT and migrating data, I expect to see all of my old data and be able to add new tasks.

Possible Solution

No response

Context

cvat_db logs:
2024-01-19 17:39:34.155 UTC [22] LOG: checkpoint complete: wrote 84 buffers (0.5%); 0 WAL file(s) added, 0 removed, 0 recycled; write=8.082 s, sync=0.134 s, total=8.330 s; sync files=60, longest=0.086 s, average=0.003 s; distance=320 kB, estimate=320 kB
2024-01-19 17:44:25.239 UTC [22] LOG: checkpoint starting: time
2024-01-19 17:44:28.346 UTC [22] LOG: checkpoint complete: wrote 30 buffers (0.2%); 0 WAL file(s) added, 0 removed, 0 recycled; write=2.845 s, sync=0.120 s, total=3.107 s; sync files=29, longest=0.105 s, average=0.005 s; distance=78 kB, estimate=296 kB
2024-01-19 17:49:25.379 UTC [22] LOG: checkpoint starting: time

cvat_server logs:
AFTER UI:
2024-01-19 19:24:21,688 DEBG 'uvicorn-0' stdout output:
INFO: 10.202.16.142:0 - "GET /api/projects/36 HTTP/1.0" 200 OK

2024-01-19 19:24:21,868 DEBG 'uvicorn-1' stdout output:
INFO: 10.202.16.142:0 - "GET /api/labels?project_id=36&org=&page_size=500&page=1 HTTP/1.0" 200 OK

2024-01-19 19:24:22,194 DEBG 'uvicorn-1' stdout output:
INFO: 10.202.16.142:0 - "POST /api/tasks?org= HTTP/1.0" 201 Created

2024-01-19 19:24:22,385 DEBG 'uvicorn-1' stdout output:
INFO: 10.202.16.142:0 - "POST /api/tasks/1167/data?org= HTTP/1.0" 202 Accepted

2024-01-19 19:24:22,680 DEBG 'uvicorn-0' stdout output:
INFO: 10.202.16.142:0 - "POST /api/events?org= HTTP/1.0" 201 Created

2024-01-19 19:24:22,861 DEBG 'uvicorn-1' stdout output:
INFO: 10.202.16.142:0 - "POST /api/tasks/1167/data?org= HTTP/1.0" 200 OK

2024-01-19 19:24:23,228 DEBG 'uvicorn-1' stdout output:
INFO: 10.202.16.142:0 - "POST /api/tasks/1167/data?org= HTTP/1.0" 200 OK

2024-01-19 19:24:23,794 DEBG 'uvicorn-1' stdout output:
INFO: 10.202.16.142:0 - "POST /api/tasks/1167/data?org= HTTP/1.0" 200 OK

2024-01-19 19:24:24,394 DEBG 'uvicorn-1' stdout output:
INFO: 10.202.16.142:0 - "POST /api/tasks/1167/data?org= HTTP/1.0" 200 OK

2024-01-19 19:24:24,786 DEBG 'uvicorn-1' stdout output:
INFO: 10.202.16.142:0 - "POST /api/tasks/1167/data?org= HTTP/1.0" 200 OK

2024-01-19 19:24:25,314 DEBG 'uvicorn-0' stdout output:
INFO: 172.28.0.2:0 - "GET /api/auth/rules HTTP/1.0" 304 Not Modified

2024-01-19 19:24:25,585 DEBG 'uvicorn-1' stdout output:
INFO: 10.202.16.142:0 - "POST /api/tasks/1167/data?org= HTTP/1.0" 200 OK

2024-01-19 19:24:26,119 DEBG 'uvicorn-1' stdout output:
INFO: 10.202.16.142:0 - "POST /api/tasks/1167/data?org= HTTP/1.0" 200 OK

2024-01-19 19:24:26,610 DEBG 'uvicorn-0' stdout output:
INFO: 10.202.16.142:0 - "POST /api/tasks/1167/data?org= HTTP/1.0" 200 OK

2024-01-19 19:24:27,018 DEBG 'uvicorn-0' stdout output:
INFO: 10.202.16.142:0 - "POST /api/tasks/1167/data?org= HTTP/1.0" 200 OK

2024-01-19 19:24:27,418 DEBG 'uvicorn-0' stdout output:
INFO: 10.202.16.142:0 - "POST /api/tasks/1167/data?org= HTTP/1.0" 200 OK

2024-01-19 19:24:27,585 DEBG 'uvicorn-0' stdout output:
INFO: 10.202.16.142:0 - "POST /api/tasks/1167/data?org= HTTP/1.0" 200 OK

2024-01-19 19:24:27,785 DEBG 'uvicorn-0' stdout output:
INFO: 10.202.16.142:0 - "POST /api/tasks/1167/data?org= HTTP/1.0" 202 Accepted

2024-01-19 19:24:28,008 DEBG 'uvicorn-1' stdout output:
INFO: 10.202.16.142:0 - "GET /api/tasks/1167/status?org= HTTP/1.0" 200 OK

2024-01-19 19:24:28,302 DEBG 'uvicorn-0' stdout output:
INFO: 10.202.16.142:0 - "DELETE /api/tasks/1167 HTTP/1.0" 204 No Content

AFTER API COMMAND LINE:
2024-01-19 19:20:42,895 DEBG 'uvicorn-0' stdout output:
INFO: 10.202.16.142:0 - "POST /api/tasks/1166/data HTTP/1.0" 202 Accepted

2024-01-19 19:20:43,506 DEBG 'uvicorn-0' stdout output:
INFO: 10.202.16.142:0 - "GET /api/tasks/1166/status HTTP/1.0" 200 OK

2024-01-19 19:20:45,093 DEBG 'uvicorn-0' stdout output:
INFO: 10.202.16.142:0 - "GET /api/tasks/1166/status HTTP/1.0" 200 OK

AFTER RESUBMIT UPLOAD IMAGES TO SAME TASK ID:
Traceback (most recent call last):
File "/opt/venv/lib/python3.10/site-packages/asgiref/sync.py", line 534, in thread_handler
raise exc_info[1]
File "/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 42, in inner
response = await get_response(request)
File "/opt/venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 253, in _get_response_async
response = await wrapped_callback(
File "/opt/venv/lib/python3.10/site-packages/asgiref/sync.py", line 479, in call
ret: _R = await loop.run_in_executor(
File "/opt/venv/lib/python3.10/site-packages/asgiref/current_thread_executor.py", line 40, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/venv/lib/python3.10/site-packages/asgiref/sync.py", line 538, in thread_handler
return func(*args, **kwargs)
File "/opt/venv/lib/python3.10/site-packages/django/views/decorators/csrf.py", line 56, in wrapper_view
return view_func(*args, **kwargs)
File "/opt/venv/lib/python3.10/site-packages/rest_framework/viewsets.py", line 125, in view
return self.dispatch(request, *args, **kwargs)
File "/opt/venv/lib/python3.10/site-packages/rest_framework/views.py", line 509, in dispatch
response = self.handle_exception(exc)
File "/opt/venv/lib/python3.10/site-packages/rest_framework/views.py", line 469, in handle_exception
self.raise_uncaught_exception(exc)
File "/opt/venv/lib/python3.10/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
raise exc
File "/opt/venv/lib/python3.10/site-packages/rest_framework/views.py", line 506, in dispatch
response = handler(request, *args, **kwargs)
File "/home/django/cvat/apps/engine/views.py", line 1213, in data
return self.upload_data(request)
File "/home/django/cvat/apps/engine/mixins.py", line 237, in upload_data
return self.upload_finished(request)
File "/home/django/cvat/apps/engine/views.py", line 1097, in upload_finished
return _handle_upload_data(request)
File "/home/django/cvat/apps/engine/views.py", line 1027, in _handle_upload_data
self.append_files(request)
File "/usr/lib/python3.10/contextlib.py", line 79, in inner
return func(*args, **kwds)
File "/home/django/cvat/apps/engine/views.py", line 993, in append_files
self._append_upload_info_entries(client_files)
File "/home/django/cvat/apps/engine/views.py", line 943, in _append_upload_info_entries
task_data.client_files.bulk_create([
File "/opt/venv/lib/python3.10/site-packages/django/db/models/manager.py", line 87, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/opt/venv/lib/python3.10/site-packages/django/db/models/query.py", line 803, in bulk_create
returned_columns = self._batched_insert(
File "/opt/venv/lib/python3.10/site-packages/django/db/models/query.py", line 1831, in _batched_insert
self._insert(
File "/opt/venv/lib/python3.10/site-packages/django/db/models/query.py", line 1805, in _insert
return query.get_compiler(using=using).execute_sql(returning_fields)
File "/opt/venv/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1822, in execute_sql
cursor.execute(sql, params)
File "/opt/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(
File "/opt/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/opt/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 84, in _execute
with self.db.wrap_database_errors:
File "/opt/venv/lib/python3.10/site-packages/django/db/utils.py", line 91, in exit
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/opt/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 89, in _execute
return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: duplicate key value violates unique constraint "engine_clientfile_data_id_file_c9989a74_uniq"
DETAIL: Key (data_id, file)=(1074, /home/django/data/data/1074/raw/12_06_WV03_VNIR_R1C1_tile_11_25.jpg) already exists.

Environment

I am using CVAT version 2.10.0 and upgrade from a version using Postgres 10 (although I couldn't find which exact version it was).
@kgramzinski kgramzinski added the bug Something isn't working label Jan 19, 2024
@bsekachev
Copy link
Member

Hello, how can we reproduce the issue?
Also, please, check that these directories exist in cvat_worker_import docker container.

@bsekachev bsekachev added the need info Need more information to investigate the issue label Jan 29, 2024
@kgramzinski
Copy link
Author

kgramzinski commented Feb 2, 2024 •
edited
Loading

Hello, sorry for the delay.

I'm not sure exactly how to reproduce my issue, but it seems to be a problem with migrating from Postgres 10 to Postgres 15. I followed the linked instruction pages (CVAT upgrade, Postgres upgrade guides) exactly. Below are the logs from cvat_worker_import.

2024-02-02 14:38:40,901 DEBG 'rqworker-import-1' stderr output:
[2024-02-02 14:38:40,900] DEBUG rq.worker: Sent heartbeat to prevent worker timeout. Next one should arrive in 480 seconds.

2024-02-02 14:38:40,902 DEBG 'rqworker-import-1' stderr output:
[2024-02-02 14:38:40,902] DEBUG rq.worker: *** Listening on import...

2024-02-02 14:38:40,903 DEBG 'rqworker-import-1' stderr output:
[2024-02-02 14:38:40,902] DEBUG rq.worker: Sent heartbeat to prevent worker timeout. Next one should arrive in 480 seconds.

2024-02-02 14:38:40,903 DEBG 'rqworker-import-1' stderr output:
[2024-02-02 14:38:40,903] DEBUG rq.worker: Dequeueing jobs on queues import and timeout 405

2024-02-02 14:40:57,995 DEBG 'rqworker-import-0' stderr output:
[2024-02-02 14:40:57,994] DEBUG rq.worker: Dequeued job create:task.id1167 from import

2024-02-02 14:40:57,995 DEBG 'rqworker-import-0' stderr output:
[2024-02-02 14:40:57,995] INFO rq.worker: import: cvat.apps.engine.task._create_thread(1167, {'chunk_size': None, 'size': 0, 'image_quality': 70, 'start_frame': 0, 'sto...) (create:task.id1167)

2024-02-02 14:40:57,996 DEBG 'rqworker-import-0' stderr output:
[2024-02-02 14:40:57,996] DEBUG rq.worker: Sent heartbeat to prevent worker timeout. Next one should arrive in 480 seconds.

2024-02-02 14:40:58,021 DEBG 'rqworker-import-0' stderr output:
[2024-02-02 14:40:58,020] DEBUG rq.worker: Started Job Registry set.

2024-02-02 14:40:58,022 DEBG 'rqworker-import-0' stderr output:
[2024-02-02 14:40:58,021] DEBUG rq.worker: Preparing for execution of Job ID create:task.id1167

2024-02-02 14:40:58,023 DEBG 'rqworker-import-0' stderr output:
[2024-02-02 14:40:58,022] DEBUG rq.worker: Sent heartbeat to prevent worker timeout. Next one should arrive in 90 seconds.

2024-02-02 14:40:58,029 DEBG 'rqworker-import-0' stderr output:
[2024-02-02 14:40:58,028] DEBUG rq.worker: Job preparation finished.

2024-02-02 14:40:58,029 DEBG 'rqworker-import-0' stderr output:
[2024-02-02 14:40:58,029] DEBUG rq.worker: Performing Job...

2024-02-02 14:40:58,058 DEBG 'rqworker-import-0' stderr output:
[2024-02-02 14:40:58,058] INFO cvat.apps.engine.task: create task #1167

2024-02-02 14:40:58,074 DEBG 'rqworker-import-0' stderr output:
[2024-02-02 14:40:58,074] DEBUG rq.worker: Job create:task.id1167 raised an exception.

2024-02-02 14:40:58,082 DEBG 'rqworker-import-0' stderr output:
[2024-02-02 14:40:58,082] DEBUG rq.worker: Handling failed execution of job create:task.id1167

2024-02-02 14:40:58,087 DEBG 'rqworker-import-0' stderr output:
[2024-02-02 14:40:58,087] DEBUG rq.worker: Handling exception for create:task.id1167.

2024-02-02 14:40:58,088 DEBG 'rqworker-import-0' stderr output:
[2024-02-02 14:40:58,088] ERROR rq.worker: [Job create:task.id1167]: exception raised while executing (cvat.apps.engine.task._create_thread)
Traceback (most recent call last):
File "/opt/venv/lib/python3.10/site-packages/rq/worker.py", line 1428, in perform_job
rv = job.perform()
File "/opt/venv/lib/python3.10/site-packages/rq/job.py", line 1278, in perform
self._result = self._execute()
File "/opt/venv/lib/python3.10/site-packages/rq/job.py", line 1315, in _execute
result = self.func(*self.args, **self.kwargs)
File "/usr/lib/python3.10/contextlib.py", line 79, in inner
return func(*args, **kwds)
File "/home/django/cvat/apps/engine/task.py", line 919, in _create_thread
w, h = extractor.get_image_size(0)
File "/home/django/cvat/apps/engine/media_extractors.py", line 485, in get_image_size
image = (next(iter(self)))[0]
File "/home/django/cvat/apps/engine/media_extractors.py", line 421, in iter
with self._get_av_container() as container:
File "/home/django/cvat/apps/engine/media_extractors.py", line 448, in _get_av_container
return av.open(self._source_path[0])
File "av/container/core.pyx", line 401, in av.container.core.open
File "av/container/core.pyx", line 272, in av.container.core.Container.cinit
File "av/container/core.pyx", line 292, in av.container.core.Container.err_check
File "av/error.pyx", line 336, in av.error.err_check
av.error.FileNotFoundError: [Errno 2] No such file or directory: '/home/django/data/data/1077/raw/20190528_141218.mp4'

2024-02-02 14:40:58,091 DEBG 'rqworker-import-0' stderr output:
[2024-02-02 14:40:58,088] DEBUG rq.worker: Invoking exception handler <function rq_exception_handler at 0x7fba0fd5a560>

2024-02-02 14:40:58,092 DEBG 'rqworker-import-0' stderr output:
[2024-02-02 14:40:58,092] DEBUG rq.worker: Invoking exception handler <function handle_rq_exception at 0x7fbab8bd6170>

2024-02-02 14:40:58,119 DEBG 'rqworker-import-0' stderr output:
[2024-02-02 14:40:58,119] DEBUG rq.worker: Sent heartbeat to prevent worker timeout. Next one should arrive in 480 seconds.

2024-02-02 14:40:58,120 DEBG 'rqworker-import-0' stderr output:
[2024-02-02 14:40:58,119] DEBUG rq.worker: Sent heartbeat to prevent worker timeout. Next one should arrive in 480 seconds.

2024-02-02 14:40:58,121 DEBG 'rqworker-import-0' stderr output:
[2024-02-02 14:40:58,121] DEBUG rq.worker: *** Listening on import...

2024-02-02 14:40:58,122 DEBG 'rqworker-import-0' stderr output:
[2024-02-02 14:40:58,121] DEBUG rq.worker: Sent heartbeat to prevent worker timeout. Next one should arrive in 480 seconds.

2024-02-02 14:40:58,122 DEBG 'rqworker-import-0' stderr output:
[2024-02-02 14:40:58,122] DEBUG rq.worker: Dequeueing jobs on queues import and timeout 405

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working need info Need more information to investigate the issue
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@bsekachev @kgramzinski