Azure Cosmos Emulator docker image is broken

[soenneker]

The Azure Cosmos DB emulator docker image is broken

Hi Azure Cosmos team,
Hopefully this can get some visibility and reach the people who can fix this.

I filed the first bug on this almost two years ago:
microsoft/azure-pipelines-tasks#15865

To boil it down: on Ubuntu 20 and later, agents that have an Intel processor fail to start with the error: This is an evaluation version. There are [165] days left in the evaluation period. Ubuntu 18 worked, but those images are no longer available on the major platforms. Since an action may or may not receive an agent with an Intel processor, the error doesn't occur 100% of the time.

This issue is currently blocking anyone who wants to use the emulator in a Linux action.

I know that the Windows runners have the emulator built into them, but one issue is containers on Windows aren't able to be run within GitHub. I know that we can run a custom agent, but that shouldn't be necessary.

Please, let me know how I can help!

Here are many, many issues talking about this problem, spanning all types of projects, stackoverflow, etc... This isn't an exhaustive list, there are so many more:

#85
#84
#81
#77
#66
#64
#60
#45
#87
#56

https://github.com/MicrosoftDocs/visualstudio-docs/issues/9970
Azure/azure-cosmos-dotnet-v3#3010
actions/runner-images#5036
actions/runner-images#6111

https://learn.microsoft.com/en-us/answers/questions/1089628/azure-cosmos-emulator-the-evaluation-period-has-ex
https://stackoverflow.com/questions/68187668/cosmos-db-emulator-for-linux-docker-this-is-an-evaluation-version-there-are-1
https://stackoverflow.com/questions/50333215/cosmos-db-emulator
https://stackoverflow.com/questions/74440386/azure-cosmos-db-emulator-linux-image-does-not-start-error-the-evaluation-perio
https://stackoverflow.com/questions/76282676/health-check-failing-for-cosmos-db-emulator
https://learn.microsoft.com/en-us/answers/questions/1088903/unable-to-start-cosmosdb-emulator-for-linux-docker
https://learn.microsoft.com/en-us/answers/questions/1393449/while-installing-cosmosdb-emulator-locally-with-do

Tagging to increase visibility:
@ealsur @sajeetharan @TheovanKraay @markjbrown @jaydestro @suvishodcitus @satyanadella

[kirankumarkolli]

@sajeetharan, @v1k1 can you please look into it?
Also please feel free to move the issue to the right appropriate target repo/place.

[sajeetharan]

@soenneker

Thank you for highlighting the issues and we apologize for the inconvenience. The Linux emulator has been in an extended preview, but we've recently addressed crucial bugs, including support for Cosmos DB for Mongo API, in our 2023 November release.

Regarding the Ubuntu 20 support, We are looking forward to upgrading the version and will update this forum soon on the dates and our progress to all the valuable asks.

We acknowledge the importance of a seamless developer experience and are actively working to improve. Enhancing the emulator is a top priority for us as we strive to deliver a superior developer-first database experience. Your feedback is invaluable, and we are committed to meeting and exceeding expectations.

[soenneker]

Thanks @sajeetharan

Please put a priority on this as a team; we're doing all sorts of non-optimal solutions to get our tests going with Cosmos, and it's souring our perception of Cosmos.

If you could let us know an estimated date when you'll be able to address this that would be greatly appreciated.

[ghost]

Just wanted to add that we are dealing with so many issues here as well, the amount of time and money that has been burned due to issues with this emulator is starting to make us look for alternatives to CosmosDB, just because the CI / CD situation is not really solvable with CosmosDB.

[pacorreia]

Want to add on top this issue, the startup script for docker container really needs a re-work, I had to take it, place my fizes and improvements to make the self-signed certificate actually match the exact desired hostname, rather than ips or localhost.
Also having kind of a luancher/wrapper to launch a windows binay does not seem the right way to do it.

Logging is also missing, we can't see what's going on if it can't start properly.

I do hope that when taking the next fixes, the team can stop, re-think it as if they were actually using it like us and provide the flexibility we need when working locally.

I'm happy to provide further details to Product Group if desired.

[zacutti]

Same here, is frustating trying to engage the team to use Cosmos Db.
Im getting this error, the emulator sometimes start but its unusable.

This is an evaluation version. There are [83] days left in the evaluation period.
Starting
Started 1/11 partitions
Started 2/11 partitions
Started 3/11 partitions
Started 4/11 partitions
Started 5/11 partitions
Started 6/11 partitions
Started 7/11 partitions
Started 8/11 partitions
Started 9/11 partitions
Started 10/11 partitions
Started 11/11 partitions
Started
This program has encountered a fatal error and cannot continue running at Mon Feb 5 21:21:45 2024
The following diagnostic information is available:

     Reason: OS Fatal Error (0x00000006)
    Message: Last Chance
    Address: 0x3fff84111082
 Parameters: 0x3fff8855cf0b
Stack Trace:
             file://package1/windows/system32/sqlpal.dll+0x0000000000298F1C
             file://package1/windows/system32/sqlpal.dll+0x0000000000296EC1
             file://package1/windows/system32/sqlpal.dll+0x0000000000266430
             file://package1/windows/system32/sqlpal.dll+0x00000000002660ED
             file://package1/windows/system32/sqlpal.dll+0x00000000003944F3
             file:///windows/System32/Drivers/netio.sys+0x0000000000001082
             file:///windows/System32/Drivers/Http.sys+0x0000000000069526
             file:///windows/System32/Drivers/Http.sys+0x000000000006AAA4
             file:///windows/System32/Drivers/Http.sys+0x0000000000004B79
             file:///windows/System32/Drivers/Http.sys+0x00000000000A3E27
             file://package1/windows/system32/sqlpal.dll+0x000000000024A57C
             file://package1/windows/system32/sqlpal.dll+0x0000000000394491
             <unknown>+0x00000000B1D45C00
    Process: 20 - cosmosdb-emulator
     Thread: 274 (application thread 0x114)
Instance Id: 063d91f5-bdcf-488b-a9bb-884fd9ece736
   Crash Id: 00d7a76c-b2bb-40df-8213-addeb5e3a9b9
Build stamp: (null)

Distribution: Ubuntu 20.04.6 LTS
Processors: 8
Total Memory: 16707002368 bytes
Timestamp: Mon Feb 5 21:21:45 2024
Last errno: -34938881
Last errno text: Unknown error -34938881
*********** PAL PANIC CORE DUMP GENERATION FAILED **********
Unable to locate handle-crash.sh. Error: File: signals.cpp:483 [Status: 0xC0000034 Object name not found errno = 0x2(2) No such file or directory]
Executing: /usr/local/bin/cosmos/handle-crash.sh with parameters
handle-crash.sh
/usr/local/bin/cosmos/cosmosdb-emulator
20
/usr/local/bin/cosmos
/tmp/cosmos/appdata/log/

 063d91f5-bdcf-488b-a9bb-884fd9ece736
 00d7a76c-b2bb-40df-8213-addeb5e3a9b9

[soenneker]

@sajeetharan Any update here? It's been ~3 weeks. Thanks

[soenneker]

@sajeetharan Checking in on any updates, thank you

[soenneker]

@sajeetharan An update would be appreciated; we're almost at the 3 month mark here with radio silence from Microsoft. Just giving us a ballpark would be helpful...

Tagging again hopefully this can get some attention
@ealsur @TheovanKraay @markjbrown @jaydestro @suvishodcitus @satyanadella

[adamzest]

@sajeetharan An update would be appreciated; we're almost at the 3 month mark here with radio silence from Microsoft. Just giving us a ballpark would be helpful...

Tagging again hopefully this can get some attention @ealsur @TheovanKraay @markjbrown @jaydestro @suvishodcitus @satyanadella

@sajeetharan any news please? Another month with our local test deployment crippled.

[docHologram]

We are unable to run mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest because once again, the evaluation period has expired. Why is there an evaluation period at all?

[kntajus]

We are unable to run mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest because once again, the evaluation period has expired. Why is there an evaluation period at all?

A new version was released about 3 weeks ago, make sure you've pulled down the latest image.

[pacorreia]

TBF this doesn't make sense anymore, to have an evaluation period.

The image already limits the RU/s, no one would use as live systemgiven it's limitations, better to remove that obsolete evaluation period and spare both sides with annoyances.

[rlarno]

Attempted to set it up using Aspire (@davidfowl) , and while Aspire does set up the emulator and the CosmosDB connectionstring correctly, the emulater is soo slow to spin up that the depending applications are unable to connect. And they only report the following error:

Unhandled exception. System.AggregateException: One or more errors occurred. (The SSL connection could not be established, see inner exception.)
60
---> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
61
---> System.IO.IOException: Received an unexpected EOF or 0 bytes from the transport stream.

So one finds a lot of hacks and old information about how to avoid the SSL Connection issues, while it really just boils down to the emulator taking way too much time to start up (on my machine close to a minute) to be of any use in an Aspire hosted solution.

This was not an exercise in falling into the pit of success.

https://stackoverflow.com/q/78648019/45045

[UnstoppableMango]

This is about par for the course with my experience working in Azure. Hit a snag following a tutorial in the docs, google the issue, find numerous multi-year old bug reports painstakingly detailing the problem with one or two replies from a Microsoft employee around when the issue was created and then radio silence. This issue, referencing a two year old issue, is itself now 7 months old. It's impossible to start anything without spending a day or two scouring the internet for the "quirks" of whatever product you want to use.

• Azure functions core tools generates a second .csproj on build, and subsequently fails to build because there are now two .csproj files.
• Azure static web apps offers out of the box auth except you don't get an access token to make authenticated calls.
• Azure pipelines won't let you use variables in azureSubscription since security verification runs before yaml evaluation, yet there is no documentation about this restriction.
• In the Azure Resource Manager Template Reference documentation, it is incredibly likely that description for any given field is effectively just the name of the field.

I went to go find an example for that point and the first resource I randomly clicked into had this gem

• I still have no idea why the property to enable Linux App Service Plans is called reserved, but at least the documentation clearly states what it does.
• The message for some errors will display with JSON in the Azure portal, but these messages are also occasionally returned programmatically which leads to nested JSON that severely decreases its readability.

Here's some I got today while trying to deploy a real Cosmos instance in Azure since I can't run the emulator on my machine

Service returned an error. Status=404 Code="NotFound" Message="Message: {\"code\":\"NotFound\",\"message\":\"Message: {\\\"Errors\\\":[\\\"Resource Not Found. Learn more: https:\\\\/\\\\/aka.ms\\\\/cosmosdb-tsg-not-found\\\"]}\\r\\nActivityId: ...

• Tags for any Microsoft hosted Docker image are hidden in the deepest darkest corners of the internet. It felt like the gates of heaven opened up when I discovered the mcr.microsoft.com web UI, yet I STILL can't seem to find the tags for this image.

The Azure pipelines issue is a special one for me, it was something I encountered shortly after graduating and starting at my first job. It's been open for my entire career as a developer and to this day, the documentation for this field still says "Specifies the Azure Resource Manager subscription connection for the deployment".

Just to add something at least remotely productive to this thread, I encountered this today while attempting to learn Cosmos DB. I develop solely on Linux so it seems I need to deploy an instance if I want to use Cosmos at all. I believe I've experienced the same as all of the others reporting this but just in case I'll include my setup below, pulled straight from the docs that claim to have been updated on "06/20/2024".

compose.yaml

services:
  cosmos:
    container_name: jdm_cosmos
    image: mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest
    ports:
      - '8081:8081'
      - '10250-10255:10250-10255'

Logs

jdm_cosmos  | This is an evaluation version.  There are [67] days left in the evaluation period.
jdm_cosmos  | Starting
jdm_cosmos  | This program has encountered a fatal error and cannot continue running at Wed Aug  7 01:26:34 2024
jdm_cosmos  | The following diagnostic information is available:
jdm_cosmos  | 
jdm_cosmos  |          Reason: Fatal Signal (0x00000001)
jdm_cosmos  |          Signal: SIGABRT - Aborted (6)
jdm_cosmos  |           Stack:
jdm_cosmos  |                  SP               IP               Function
jdm_cosmos  |                  ---------------- ---------------- ----------------
jdm_cosmos  |                  0000708acd1b8ea0 000060c823520bfa <unknown>
jdm_cosmos  |                  0000708acd1b9f00 000060c8235205cf <unknown>
jdm_cosmos  |                  0000708acd1ba160 000060c82351fa61 <unknown>
jdm_cosmos  |                  0000708acd1ba180 0000708ae15a1090 killpg+0x40
jdm_cosmos  |                  0000708acd1bae60 0000708ae15a100b gsignal+0xcb
jdm_cosmos  |                  0000708acd1baf80 0000708ae1580859 abort+0x12b
jdm_cosmos  |                  0000708acd1bb0b0 000060c8234b8332 <unknown>
jdm_cosmos  |                  0000708acd1bb160 000060c82353a667 <unknown>
jdm_cosmos  | 
jdm_cosmos  |         Process: 20 - cosmosdb-emulator
jdm_cosmos  |          Thread: 125 (application thread 0x1c0)
jdm_cosmos  |     Instance Id: 3949a854-9341-4eac-947d-48301e7142b1
jdm_cosmos  |        Crash Id: c1a2dcac-abeb-4ae3-9ace-bdd2c13d4488
jdm_cosmos  |     Build stamp: (null)
jdm_cosmos  |    Distribution: Ubuntu 20.04.6 LTS
jdm_cosmos  |      Processors: 16
jdm_cosmos  |    Total Memory: 67263025152 bytes
jdm_cosmos  |       Timestamp: Wed Aug  7 01:26:34 2024
jdm_cosmos  |      Last errno: -34938881
jdm_cosmos  | Last errno text: Unknown error -34938881
jdm_cosmos  | *********** PAL PANIC CORE DUMP GENERATION FAILED **********
jdm_cosmos  | Unable to locate handle-crash.sh. Error: File: signals.cpp:483 [Status: 0xC0000034 Object name not found errno = 0x2(2) No such file or directory]
jdm_cosmos  | *********** PANIC CORE DUMP GENERATION FAILED **********
jdm_cosmos  | Attempt to launch handle-crash.sh failed.
jdm_cosmos  | Executing: /usr/local/bin/cosmos/handle-crash.sh with parameters
jdm_cosmos  |      handle-crash.sh
jdm_cosmos  |      /usr/local/bin/cosmos/cosmosdb-emulator
jdm_cosmos  |      20
jdm_cosmos  |      /usr/local/bin/cosmos
jdm_cosmos  |      /tmp/cosmos/appdata/log/
jdm_cosmos  |      
jdm_cosmos  |      3949a854-9341-4eac-947d-48301e7142b1
jdm_cosmos  |      c1a2dcac-abeb-4ae3-9ace-bdd2c13d4488
jdm_cosmos  |      
jdm_cosmos  | This program has encountered a fatal error and cannot continue running at Wed Aug  7 01:26:34 2024
jdm_cosmos  | The following diagnostic information is available:
jdm_cosmos  | 
jdm_cosmos  |          Reason: Host Extension RTL_ASSERT (0x00000003)
jdm_cosmos  |          Status: STATUS_WAIT_2 (0x00000002)
jdm_cosmos  |         Message: !killTheTarget
jdm_cosmos  |           Stack:
jdm_cosmos  |                  SP               IP               Function
jdm_cosmos  |                  ---------------- ---------------- ----------------
jdm_cosmos  |                  0000708adf5fbdf0 000060c823520bfa <unknown>
jdm_cosmos  |                  0000708adf5fce50 000060c8235205cf <unknown>
jdm_cosmos  |                  0000708adf5fd0b0 000060c82348ab66 <unknown>
jdm_cosmos  |                  0000708adf5fd0e0 000060c8235240e4 <unknown>
jdm_cosmos  |                  0000708adf5fe310 000060c8235237a9 <unknown>
jdm_cosmos  |                  0000708adf5fe440 0000708ae1bfc609 start_thread+0xd9
jdm_cosmos  |                  0000708adf5fe500 0000708ae167d353 clone+0x43
jdm_cosmos  | 
jdm_cosmos  |         Process: 18 - cosmosdb-emulator
jdm_cosmos  |          Thread: 19
jdm_cosmos  |     Instance Id: 3949a854-9341-4eac-947d-48301e7142b1
jdm_cosmos  |        Crash Id: c1a2dcac-abeb-4ae3-9ace-bdd2c13d4488
jdm_cosmos  |     Build stamp: (null)
jdm_cosmos  |    Distribution: Ubuntu 20.04.6 LTS
jdm_cosmos  |      Processors: 16
jdm_cosmos  |    Total Memory: 67263025152 bytes
jdm_cosmos  |       Timestamp: Wed Aug  7 01:26:34 2024
jdm_cosmos  |      Last errno: 2
jdm_cosmos  | Last errno text: No such file or directory
jdm_cosmos  | Aborted (core dumped)
jdm_cosmos exited with code 134

[Hennrikandreassonha]

Any updates on this?

[HasanKasamali]

Any updates on this please. I am getting the same error as 'UnstoppableMango' is getting (39 days left in evaluation period, program encountered a fatal error, panic core dump and all the other errors).

I have the latest version of docker desktop installed on windows 10 professional and latest docker engine on Ubuntu 24.04.

The same error occurs on both

1. Linux container in Windows 10 professional and

2. Ubuntu 24.04 LTS.

3. For windows 10 professional I am just using the Run button in Docker Desktop to start the emulator.

4. For Ubuntu 24.04 I am following the below Microsoft link as a reference and launching the emulator in docker container from the terminal:

Develop locally using the Azure Cosmos DB emulator (update on 14 Aug 2024)
https://learn.microsoft.com/en-us/azure/cosmos-db/how-to-develop-emulator?tabs=docker-linux%2Ccsharp&pivots=api-nosql

Many Thanks in advance

[belzano]

Any updates on this?

[diegosasw]

I'm afraid It won't be fixed. It's been like this for years.
I've spent many many hours running tests, building sandboxes to use the CosmosDb emulator docker image (like on this repo where I use .NET and TestContainers)
and, while it works for simple scenarios, as soon as you start adding more load on CosmosDb, it breaks, it starts giving Too Many Requests errors, Server Errors, etc.

[belzano]

That's my understanding too.
Switch to the MongoDb driver allows a mongodb container instance usage for local/integration.
Added benefit: Our application is not tied to the Azure ecosystem, or at the mercy of a CosmosDb end of support.

[alanta]

The new Linux-based emulator looks promising https://devblogs.microsoft.com/cosmosdb/introducing-the-new-linux-based-azure-cosmos-db-emulator-preview/

Starts quite quickly on my machine.

[Sabrina3f]

Yeah but how to get the mondodb api working the vnext-preview?