Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Linux container fails to start up with SIGABRT #84

Closed
szszoke opened this issue Jan 23, 2024 · 10 comments
Closed

Linux container fails to start up with SIGABRT #84

szszoke opened this issue Jan 23, 2024 · 10 comments
Assignees
@sajeetharan @amanrao23
Labels
bug
Milestone
Feb2024

Comments

@szszoke
Copy link

szszoke commented Jan 23, 2024 •
edited
Loading

Describe the bug
The Linux container fails to start up with SIGABRT.

To Reproduce
Steps to reproduce the behavior:

docker run \                       
    --publish 8081:8081 \
    --publish 10250-10255:10250-10255 \
    --interactive \
    --tty \
    mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest

Expected behavior
The container should start up without errors.

Screenshots

$ docker run \                       
    --publish 8081:8081 \
    --publish 10250-10255:10250-10255 \
    --interactive \
    --tty \
    mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest
Unable to find image 'mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest' locally
latest: Pulling from cosmosdb/linux/azure-cosmos-emulator
d7bfe07ed847: Pull complete 
520d4c735f8c: Pull complete 
0722c4f37e97: Pull complete 
f1a6e15ab569: Pull complete 
6f9ffa3152bf: Pull complete 
cc3063780407: Pull complete 
14cd358b5760: Pull complete 
dfaba22a8450: Pull complete 
5a96b88a19ba: Pull complete 
f4133c361e5f: Pull complete 
85c85b80785c: Pull complete 
74e2278670c0: Pull complete 
3e169396e112: Pull complete 
6291d3a77d8a: Pull complete 
4f4fb700ef54: Pull complete 
712692508376: Pull complete 
1b4ec4f9b41f: Pull complete 
86e8e6ee474c: Pull complete 
75e82f426270: Pull complete 
a26d85cd5ccf: Pull complete 
Digest: sha256:85c7b1771de120d3923c077606383e52b3763a7fe740c9f13bb2a95b5cb841e9
Status: Downloaded newer image for mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest
This is an evaluation version.  There are [96] days left in the evaluation period.
Starting
This program has encountered a fatal error and cannot continue running at Tue Jan 23 10:13:34 2024
The following diagnostic information is available:

         Reason: Fatal Signal (0x00000001)
         Signal: SIGABRT - Aborted (6)
          Stack:
                 SP               IP               Function
                 ---------------- ---------------- ----------------
                 00007b28b1b76fa0 00005d8b4ef95bfa <unknown>
                 00007b28b1b78000 00005d8b4ef955cf <unknown>
                 00007b28b1b78260 00005d8b4ef94a61 <unknown>
                 00007b28b1b78280 00007b28b61eb090 killpg+0x40
                 00007b28b1b78e60 00007b28b61eb00b gsignal+0xcb
                 00007b28b1b78f80 00007b28b61ca859 abort+0x12b
                 00007b28b1b790b0 00005d8b4ef2eaa2 <unknown>
                 00007b28b1b79160 00005d8b4efaf657 <unknown>

        Process: 19 - cosmosdb-emulator
         Thread: 126 (application thread 0x1c8)
    Instance Id: 5a3e9f7a-43ba-4a11-9c9e-55ad1e163626
       Crash Id: 56eda8b5-e7d6-421f-8478-351b2a16042b
    Build stamp: (null)
   Distribution: Ubuntu 20.04.6 LTS
     Processors: 16
   Total Memory: 33028304896 bytes
      Timestamp: Tue Jan 23 10:13:34 2024
     Last errno: -34938881
Last errno text: Unknown error -34938881
*********** PAL PANIC CORE DUMP GENERATION FAILED **********
Unable to locate handle-crash.sh. Error: File: signals.cpp:483 [Status: 0xC0000034 Object name not found errno = 0x2(2) No such file or directory]
Executing: /usr/local/bin/cosmos/handle-crash.sh with parameters
     handle-crash.sh
     /usr/local/bin/cosmos/cosmosdb-emulator
     19
     /usr/local/bin/cosmos
     /tmp/cosmos/appdata/log/
     
     5a3e9f7a-43ba-4a11-9c9e-55ad1e163626
     56eda8b5-e7d6-421f-8478-351b2a16042b
     
*********** PANIC CORE DUMP GENERATION FAILED **********
Attempt to launch handle-crash.sh failed.
This program has encountered a fatal error and cannot continue running at Tue Jan 23 10:13:34 2024
The following diagnostic information is available:

         Reason: Host Extension RTL_ASSERT (0x00000003)
         Status: STATUS_WAIT_2 (0x00000002)
        Message: !killTheTarget
          Stack:
                 SP               IP               Function
                 ---------------- ---------------- ----------------
                 00007b28b3dfbdf0 00005d8b4ef95bfa <unknown>
                 00007b28b3dfce50 00005d8b4ef955cf <unknown>
                 00007b28b3dfd0b0 00005d8b4eeffb56 <unknown>
                 00007b28b3dfd0e0 00005d8b4ef990e4 <unknown>
                 00007b28b3dfe310 00005d8b4ef987a9 <unknown>
                 00007b28b3dfe440 00007b28b6846609 start_thread+0xd9
                 00007b28b3dfe500 00007b28b62c7133 clone+0x43

        Process: 17 - cosmosdb-emulator
         Thread: 18
    Instance Id: 5a3e9f7a-43ba-4a11-9c9e-55ad1e163626
       Crash Id: 56eda8b5-e7d6-421f-8478-351b2a16042b
    Build stamp: (null)
   Distribution: Ubuntu 20.04.6 LTS
     Processors: 16
   Total Memory: 33028304896 bytes
      Timestamp: Tue Jan 23 10:13:34 2024
     Last errno: 2
Last errno text: No such file or directory
Aborted (core dumped)

Desktop (please complete the following information):

  • OS: Arch Linux
  • Browser N/A
  • Version N/A

Docker Images Used:

  • Linux

Arguments && Environment variables to start Docker:

/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

Docker Environment

  • Standalone

Additional context

This has been working sometime near the the end of 2023.

$ docker --version
Docker version 24.0.7, build afdd53b4e3
@soenneker soenneker mentioned this issue Jan 26, 2024
Open
@sajeetharan
Copy link
Member

@szszoke can you please confirm the OS , whether it is ubuntu or arch linux?

@szszoke
Copy link
Author

szszoke commented Jan 30, 2024

The host OS running Docker itself is Arch Linux, the Cosmos DB image seems to be based on an Ubuntu image.

@sajeetharan sajeetharan added this to the Feb2024 milestone Jan 30, 2024
@max-tet
Copy link

max-tet commented Feb 1, 2024

I have the same issue, also on Arch Linux.
I'd like to use an older version of the image in the meantime because at some point in the past it did work. What are some tags I can use besides latest? I can't seem to find a list of tags in the registry.

@szszoke
Copy link
Author

szszoke commented Feb 1, 2024

The latest image hasn't been updated since 2023 November 03 and I remember this working after that.

@szszoke
Copy link
Author

szszoke commented Feb 1, 2024 •
edited
Loading

In the meantime, I found an issue on MS SQL that looked very similar and I tried to downgrade my kernel to the latest 6.6 release which fixed the issue.

microsoft/mssql-docker#868

It is quite likely the the same root cause is at play here and maybe the people responsible for Cosmos DB could talk to the people responsible for MS SQL to see if it is actually the same problem.

Here is how I downgraded my kernel if anyone needs it:

$ cd /var/cache/pacman/pkg
$ sudo pacman -U linux-6.6.10.arch1-1-x86_64.pkg.tar.zst linux-headers-6.6.10.arch1-1-x86_64.pkg.tar.zst
$ sudo reboot

You might also want to tell pacman to not update those two packages. You can do that by opening /etc/pacman.conf and adding the following line:

IgnorePkg = linux linux-headers

I still had the older package binaries on my system if you don't have them anymore then you will have to figure out how to get them yourself.

@max-tet
Copy link

max-tet commented Feb 3, 2024

This seems to have worked, thanks a lot!
Although I always fear that I break something when doing something like this. This should not be needed.

@christianeriksson
Copy link

Just want to note that I have the same issue (also on Arch, on kernel 6.7.3). The temporary workaround that szszoke provided got me up and running again (downgrade to kernel 6.6.1). Hope this gets solved soon.

@tonytheleg
Copy link

Seeing the same issue on Fedora 39 so its not just specific to Arch.

docker run \
    --publish 8081:8081 \
    --publish 10250-10255:10250-10255 \
    --interactive \
    --tty \
    mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest   
This is an evaluation version.  There are [145] days left in the evaluation period.
Starting
This program has encountered a fatal error and cannot continue running at Tue May 21 14:20:24 2024
The following diagnostic information is available:

         Reason: Fatal Signal (0x00000001)
         Signal: SIGABRT - Aborted (6)
          Stack:
                 SP               IP               Function
                 ---------------- ---------------- ----------------
                 00007f06cbec2260 000055c49b0babfa <unknown>
                 00007f06cbec32c0 000055c49b0ba5cf <unknown>
                 00007f06cbec3520 000055c49b0b9a61 <unknown>
                 00007f06cbec3540 00007f06d7ba1090 killpg+0x40
                 00007f06cbec3bd0 00007f06d7ba100b gsignal+0xcb
                 00007f06cbec3cf0 00007f06d7b80859 abort+0x12b
                 00007f06cbec3e20 000055c49b052332 <unknown>
                 00007f06cbec3ed0 000055c49b0dbb14 <unknown>
                 00007f06cbec3ef0 000055c49b10af38 <unknown>
                 00007f06cbec3fa0 000055c49b10ad1a <unknown>
                 00007f06cbec4000 000055c49b05e1ba <unknown>
                 00007f06cbec4080 000055c49b05de0d <unknown>
                 00007f06cbec4160 000055c49b0d3361 <unknown>

        Process: 19 - cosmosdb-emulator
         Thread: 126 (application thread 0x1cc)
    Instance Id: 86fef1fc-1395-4339-a1d9-121eb39f9a36
       Crash Id: e361823e-d63e-4f26-acdc-80ded371315d
    Build stamp: (null)
   Distribution: Ubuntu 20.04.6 LTS
     Processors: 8
   Total Memory: 33393545216 bytes
      Timestamp: Tue May 21 14:20:24 2024
     Last errno: -34938881
Last errno text: Unknown error -34938881
*********** PAL PANIC CORE DUMP GENERATION FAILED **********
Unable to locate handle-crash.sh. Error: File: signals.cpp:483 [Status: 0xC0000034 Object name not found errno = 0x2(2) No such file or directory]
Executing: /usr/local/bin/cosmos/handle-crash.sh with parameters
     handle-crash.sh
     /usr/local/bin/cosmos/cosmosdb-emulator
     19
     /usr/local/bin/cosmos
     /tmp/cosmos/appdata/log/
     
     86fef1fc-1395-4339-a1d9-121eb39f9a36
     e361823e-d63e-4f26-acdc-80ded371315d
     
*********** PANIC CORE DUMP GENERATION FAILED **********
Attempt to launch handle-crash.sh failed.
This program has encountered a fatal error and cannot continue running at Tue May 21 14:20:24 2024
The following diagnostic information is available:

         Reason: Host Extension RTL_ASSERT (0x00000003)
         Status: STATUS_WAIT_2 (0x00000002)
        Message: !killTheTarget
          Stack:
                 SP               IP               Function
                 ---------------- ---------------- ----------------
                 00007f06d5bfbdf0 000055c49b0babfa <unknown>
                 00007f06d5bfce50 000055c49b0ba5cf <unknown>
                 00007f06d5bfd0b0 000055c49b024b66 <unknown>
                 00007f06d5bfd0e0 000055c49b0be0e4 <unknown>
                 00007f06d5bfe310 000055c49b0bd7a9 <unknown>
                 00007f06d5bfe440 00007f06d81fc609 start_thread+0xd9
                 00007f06d5bfe500 00007f06d7c7d353 clone+0x43

        Process: 17 - cosmosdb-emulator
         Thread: 18
    Instance Id: 86fef1fc-1395-4339-a1d9-121eb39f9a36
       Crash Id: e361823e-d63e-4f26-acdc-80ded371315d
    Build stamp: (null)
   Distribution: Ubuntu 20.04.6 LTS
     Processors: 8
   Total Memory: 33393545216 bytes
      Timestamp: Tue May 21 14:20:24 2024
     Last errno: 2
Last errno text: No such file or directory
Aborted (core dumped)

@alexfandos alexfandos mentioned this issue Aug 21, 2024
Closed
@untitled-confused-goose
Copy link

To add another data point:
Booting into an older kernel works for me (older docker version did not).

On my KDE Neon machine (Ubuntu based) it fails for the "6.8.0-40-generic" kernel but works on the "6.5.0-45-generic" kernel.

@niteshvijay1995
Copy link

niteshvijay1995 commented Sep 19, 2024 •
edited
Loading

The issue is fixed in new release (2.14.20)
Pull the latest image from mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sajeetharan sajeetharan

@amanrao23 amanrao23

Labels
bug
Projects
None yet
Milestone
Feb2024
Development

No branches or pull requests
8 participants
@max-tet @szszoke @sajeetharan @niteshvijay1995 @tonytheleg @amanrao23 @untitled-confused-goose @christianeriksson