Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Linux emulator fails on macOS M1 even with x64 virtualization #54

Open
chriskuech opened this issue Jul 8, 2022 · 142 comments
Open

Linux emulator fails on macOS M1 even with x64 virtualization #54

chriskuech opened this issue Jul 8, 2022 · 142 comments
Assignees

Comments

@chriskuech
Copy link

Understandably, you do not support M1 chips and it is questionable if supporting M1 chips is worth prioritizing; however, somehow the container fails on M1 even when using x64 virtualization, as demonstrated in #17.

At a minimum, can we have the container work with x64 emulation?

@anton-bruilo
Copy link

Please do something for start it on Mac M1 ..
I see that another topics about that exists more than 1 year.

@jonas-lomholdt
Copy link

jonas-lomholdt commented Aug 3, 2022

This would be an awesome addition since we are seeing more and more M1's as dev boxes.

@tinydogio-joshua
Copy link

I would also appreciate seeing this working. 🙏

@theramzay
Copy link

2+ years past since WWDC 2020. We're all need support for Apple silicon.

@daveHylde
Copy link

daveHylde commented Oct 10, 2022

100% agree that the emulator should work on arm64. I've created this azure feedback post on the issue. Don't know if Microsoft care about it, but worth a shot.
https://feedback.azure.com/d365community/idea/dc98013b-7348-ed11-a81b-000d3ae3db6e

@chriskuech
Copy link
Author

FYI we ended up migrating off CosmosDB because of this

@niels-busch-maersk
Copy link

niels-busch-maersk commented Oct 14, 2022

FYI we ended up migrating off CosmosDB because of this

One idea could be to use the MongoDB driver with CosmosDB and then replace emulator with MongoDB image.

Don't go down that route!

@murbanowicz
Copy link

@milismsft @mominag @aliuy

@waszak
Copy link

waszak commented Oct 26, 2022

Try asking here https://learn.microsoft.com/en-us/answers/topics/azure-cosmos-db.html
Maybe someone will look on this issue

@tebeco
Copy link

tebeco commented Dec 15, 2022

maybe @jongio know who to contact or existing workaround / replacement

@jongio
Copy link
Member

jongio commented Dec 15, 2022

@markjbrown would likely know

@markjbrown
Copy link

There's no workaround for this unfortunately. This is on our roadmap to deliver but I don't have any ETA.

@tebeco
Copy link

tebeco commented Dec 15, 2022

can you point out where the source is for the MSI hosted at https://aka.ms/cosmosdb-emulator that is in the Dockerfile

maybe someone might find out how to get update (replace/remove) all the servercore specifics to make it more XPlat ?

@markjbrown
Copy link

The emulator is not open source.

@tebeco
Copy link

tebeco commented Dec 15, 2022

should it be concidered to open source it ?

if not, should all the communication protocol be open / documented or is it already ?
i'm thinking like the LSP

private impl could be kept private

It can only benefits everyone

@markjbrown
Copy link

Yes, I will definitely take that suggestion to the team.

Thank you!

@Compusa
Copy link

Compusa commented Dec 16, 2022

Please do whatever it takes to prioritise and get this resolved soon. Not having proper tool support is a show stopper.

Everything works perfectly in my development environment on the MacBook Pro M1, except the Azure Cosmos DB Emulator.

My current workaround is to run the Azure Cosmos DB Emulator on Parallels Desktop, but that workaround has a lot of disadvantages, and I wouldn't need Parallel Desktop if it wasn't for this...

@redhatpeter
Copy link

Please make sure to make it work for MacBook Pro M1. That will be a BIG help.

@irby
Copy link

irby commented Jan 17, 2023

Would love to see M1 support.

@gc-6point6
Copy link

gc-6point6 commented Jan 18, 2023

Not having M1 support is a big limitation, and it has definitely a major impact when it comes down to choosing a Cloud platform and/or Azure resource selection.

@tinydogio-joshua
Copy link

Agreed. Between this and not supporting mssql on aarch64 in docker, I'm concerned about proper development support of these products. Seems they're trying to rely on Docker supporting Rosetta (which is supposed to be a temporary/transition piece, not a something to rely on long term).

@potatoqualitee
Copy link

Oof, ran into this issue today. We need this for our Cosmos db app development as well.

@Compusa
Copy link

Compusa commented Jan 20, 2023

Oof, ran into this issue today. We need this for our Cosmos db app development as well.

Unfortunately the people responsible of this product doesn’t seem to care that much about this? Everything works just fine for me as .NET developer on a MacBook Pro M1 except the Azure Cosmos DB Emulator.

@theramzay
Copy link

Oof, ran into this issue today. We need this for our Cosmos db app development as well.

Unfortunately the people responsible of this product doesn’t seem to care that much about this? Everything works just fine for me as .NET developer on a MacBook Pro M1 except the Azure Cosmos DB Emulator.

Yep, same for me as dotnet/react developer. CosmosDB emulator the only one software on the world, that not runs at all on my arm64 Mac. It's looks insane for 2023. It not works even in latest Docker "Rosetta 2 in Linux" feature.

@Compusa
Copy link

Compusa commented Jan 20, 2023

The latest beta of Docker Desktop has a new feature for better support of running x86/64 binaries with Rosetta 2. Would this make it possible to run the Linux Emulator on M1 chips? I haven’t had the possibility to test this my self yet, so if anyone wants, please try it out and reach back here with the result please :)

More info about this can be found here:
docker/roadmap#384

@theramzay
Copy link

theramzay commented Jan 20, 2023

The latest beta of Docker Desktop has a new feature for better support of running x86/64 binaries with Rosetta 2. Would this make it possible to run the Linux Emulator on M1 chips? I haven’t had the possibility to test this my self yet, so if anyone wants, please try it out and reach back here with the result please :)

More info about this can be found here:

docker/roadmap#384

Nope. I've tested it already. Emulator Just run & die immediately after start.

@markjbrown
Copy link

Hi folks. Thanks for all your comments.

As I mentioned back in December this is on our roadmap, but I don't have an ETA.

It might help if I explain how features get committed on our team. I think most folks here understand this, but I want to take a moment to explain to help set expectations and provide some solace to those who may not think we care or paying attention to this.

The Cosmos team (along with every service team in Azure) organizes feature work in 6-month semester that starts after a month of planning where work items are committed. The current semester started in October and runs through March. Despite Microsoft's size, feature teams are very lean. Items that are committed are done after careful resource planning to ensure we don't over commit, resulting in slips or dropped items. This means that even if we see intense demand for something after planning, (like this one here where the bulk of input occurred after the semester had started), it is unlikely we could take it on. This would often require dropping other committed work which could lead to a cascade of issues for downstream dependencies.

The planning for our next semester (April-October) will start sometime near the end of February next month. M1 support is among the list of features that will be reviewed for this upcoming semester. This GitHub issue here that you have been commenting and voting the feature up on, has been filed in support of the feature and is being followed.

We recognize and appreciate your continued interest and support for this. For people who are seeing this Issue for the first time, you are welcome to add your support as well. Once planning for our upcoming semester has concluded, myself or another on our team will revert back here on the outcomes.

Thanks again.

@jonas-lomholdt
Copy link

jonas-lomholdt commented Jan 23, 2023

@markjbrown thanks for the insights on feature semesters. Don't sweat it - I'm just hoping for this to get committed to in the next semester, fingers crossed 🤞 And trying to create some buzz here to get the necessary attention 😉

@solo812
Copy link

solo812 commented Feb 5, 2023

Would really love to see this. Tis a big deal for integration tests and regular development flows.

@cjshelton
Copy link

Sorry, ignore my last comment... I see you already did that.

Image

How is this done?

Mine is set to aarch64 (m2 mbp)

Image

Colima can be started with an architecture specified, e.g. colima start --arch x86_64. See colima start --help for more info.

I don't start start mine with --arch x86_64 but it seems to start with that architecture by default. Maybe I did some configuration when I installed colima which I can't remember.

@alexjamesbrown
Copy link

Interesting...

This is what I did

brew install colima
brew install qemu

colima start \
    --arch x86_64 \
    --vm-type=vz \
    --vz-rosetta \
    --mount-type=virtiofs \
    --memory 4

docker pull mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest

docker run --publish 8081:8081 --publish 10250-10255:10250-10255 --name linux-emulator --detach mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest

Now, cosmos emulator looks like it's started (no errors about unsupported platform)
However, when I hit https://localhost:8081 it never renders... just hangs.

@cjshelton
Copy link

linux-emulator

What logs do you see if you try docker logs linux-emulator?

For me I see the following, and it also takes about 2-3 mins to become available

Image

@andrewhessler
Copy link

@cjshelton Have you hit it with anything yet?

@breaker05
Copy link

This is great! This is working on my M1 macbook pro!

Thank you @cjshelton

It did take a few minutes for it to start up as well but works!

@sajeetharan
Copy link
Member

hi @breaker05, thanks for sharing about the test you did with Colima! We tested this a few months back and wanted to gather community feedback first. Could you summarize your experience or any feedback in this discussion thread? #114

@atkinsonbg
Copy link

Oh man I want to use this so bad, but it's still failing for me:

docker run --publish 8081:8081 --publish 10250-10255:10250-10255 --platform linux/amd64 --detach mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest

This command gets me this:

<jemalloc>: MADV_DONTNEED does not work (memset will be used instead)
<jemalloc>: (This is the expected behaviour if you are running under QEMU)
./cosmosdb-emulator: ERROR: Invalid mapping of address 0x4005355000 in reserved address space below 0x400000000000. Possible causes:
1) the process (itself, or via a wrapper) starts-up its own running environment sets the stack size limit to unlimited via syscall setrlimit(2);
2) the process (itself, or via a wrapper) adjusts its own execution domain and flag the system its legacy personality via syscall personality(2);
3) sysadmin deliberately sets the system to run on legacy VA layout mode by adjusting a sysctl knob vm.legacy_va_layout.

I'm running Apple M3 Pro, Sonoma 14.7.1, Colima

@tombiddulph
Copy link

tombiddulph commented Nov 18, 2024

Interesting...

This is what I did

brew install colima
brew install qemu

colima start \
    --arch x86_64 \
    --vm-type=vz \
    --vz-rosetta \
    --mount-type=virtiofs \
    --memory 4

docker pull mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest

docker run --publish 8081:8081 --publish 10250-10255:10250-10255 --name linux-emulator --detach mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest

Now, cosmos emulator looks like it's started (no errors about unsupported platform) However, when I hit https://localhost:8081 it never renders... just hangs.

I tried this and the emulator is now running on my M1 max, I can access the explorer however most calls from the dotnet sdk timeout with 503, interestingly CosmosClient.ReadAccountAsync() does return a response.

Edit:

I got it to work by setting these 2 properties in CosmosClientOptions when insantiating the CosmosClient

  ConnectionMode = ConnectionMode.Gateway,
  LimitToEndpoint = true

@jvalkeejarvi
Copy link

jvalkeejarvi commented Nov 19, 2024

I faced the same issue as @tombiddulph . There seems to be at least couple open issues related to these 503 errors:
#68
Azure/azure-cosmos-dotnet-v2#817

@markjbrown do you know if the upcoming preview release of containerized emulator fixes these issues?

@trolund
Copy link

trolund commented Nov 19, 2024

Interesting...

This is what I did

brew install colima
brew install qemu

colima start \
    --arch x86_64 \
    --vm-type=vz \
    --vz-rosetta \
    --mount-type=virtiofs \
    --memory 4

docker pull mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest

docker run --publish 8081:8081 --publish 10250-10255:10250-10255 --name linux-emulator --detach mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest

Now, cosmos emulator looks like it's started (no errors about unsupported platform) However, when I hit https://localhost:8081 it never renders... just hangs.

This worked for me and i get the web UI on https://localhost:8081/_explorer/index.html.
My system is: Macbook pro M1, MacOS 15.1, Docker 27.3.1, Colima 0.8.0.

@tombiddulph
Copy link

I faced the same issue as @tombiddulph . There seems to be at least couple open issues related to these 503 errors: #68 Azure/azure-cosmos-dotnet-v2#817

@markjbrown do you know if the upcoming preview release of containerized emulator fixes these issues?

If you're using dotnet you need to set these properties in your cosmos client

  ConnectionMode = ConnectionMode.Gateway,
  LimitToEndpoint = true

If you're using another sdk you need to set the equivalent in that.

@markjbrown
Copy link

Thanks @tombiddulph !!!

@jvalkeejarvi
Copy link

I faced the same issue as @tombiddulph . There seems to be at least couple open issues related to these 503 errors: #68 Azure/azure-cosmos-dotnet-v2#817
@markjbrown do you know if the upcoming preview release of containerized emulator fixes these issues?

If you're using dotnet you need to set these properties in your cosmos client

ConnectionMode = ConnectionMode.Gateway,
LimitToEndpoint = true
If you're using another sdk you need to set the equivalent in that.

Thanks! Have you had any 503 or 408 errors after setting these properties?

@TheovanKraay
Copy link
Member

Hello all - please try out the new linux-based emulator, it has been released today in preview, and will run on macOS M1 and other previously unsupported platforms - see readme right here in this repo. See also here for feature support. If you find issues or it doesn't support a feature that is important to you, please raise an issue in this repo, and tag it with cosmosEmulatorVnextPreview. Thanks!

@tympaniplayer
Copy link

Hello all - please try out the new linux-based emulator, it has been released today in preview, and will run on macOS M1 and other previously unsupported platforms - see readme right here in this repo. See also here for feature support. If you find issues or it doesn't support a feature that is important to you, please raise an issue in this repo, and tag it with cosmosEmulatorVnextPreview. Thanks!

It works! I'm so happy I could cry!

@trolund
Copy link

trolund commented Nov 20, 2024

Hello all - please try out the new linux-based emulator, it has been released today in preview, and will run on macOS M1 and other previously unsupported platforms - see readme right here in this repo. See also here for feature support. If you find issues or it doesn't support a feature that is important to you, please raise an issue in this repo, and tag it with cosmosEmulatorVnextPreview. Thanks!

I get this error with the new image when i try to go to http://localhost:8081/_explorer:

{
  "code": "InternalServerError",
  "message": "System.NullReferenceException: Object reference not set to an instance of an object.\n   at Microsoft.Azure.Cosmos.Postgres.Core.Interop.SqlMessageFormatter.IsRootPath(SqlRequest request) in /tmp/gateway/Cosmos.Postgres.Core/Interop/SqlMessageFormatter.cs:line 767\n   at Microsoft.Azure.Cosmos.Postgres.Core.Interop.SqlMessageFormatter.FormatRequest(KestrelHttpRequestContext transportRequestContext) in /tmp/gateway/Cosmos.Postgres.Core/Interop/SqlMessageFormatter.cs:line 117\n   at Microsoft.Azure.Cosmos.Postgres.Core.Interop.SqlRequestPipeline.ProcessRequestAsync(KestrelHttpRequestContext transportRequest) in /tmp/gateway/Cosmos.Postgres.Core/Interop/SqlRequestPipeline.cs:line 46"
}

@UberAtziri
Copy link

UberAtziri commented Nov 20, 2024

Hello all - please try out the new linux-based emulator, it has been released today in preview, and will run on macOS M1 and other previously unsupported platforms - see readme right here in this repo. See also here for feature support. If you find issues or it doesn't support a feature that is important to you, please raise an issue in this repo, and tag it with cosmosEmulatorVnextPreview. Thanks!

I get this error with the new image when i try to go to http://localhost:8081/_explorer:

{
"code": "InternalServerError",
"message": "System.NullReferenceException: Object reference not set to an instance of an object.\n at Microsoft.Azure.Cosmos.Postgres.Core.Interop.SqlMessageFormatter.IsRootPath(SqlRequest request) in /tmp/gateway/Cosmos.Postgres.Core/Interop/SqlMessageFormatter.cs:line 767\n at Microsoft.Azure.Cosmos.Postgres.Core.Interop.SqlMessageFormatter.FormatRequest(KestrelHttpRequestContext transportRequestContext) in /tmp/gateway/Cosmos.Postgres.Core/Interop/SqlMessageFormatter.cs:line 117\n at Microsoft.Azure.Cosmos.Postgres.Core.Interop.SqlRequestPipeline.ProcessRequestAsync(KestrelHttpRequestContext transportRequest) in /tmp/gateway/Cosmos.Postgres.Core/Interop/SqlRequestPipeline.cs:line 46"
}

Same here. I try to open http://localhost:8081/_explorer/index.html on M3 CPU.

{
  "code": "InternalServerError",
  "message": "System.NullReferenceException: Object reference not set to an instance of an object.\n   at Microsoft.Azure.Cosmos.Postgres.Core.Interop.SqlMessageFormatter.IsRootPath(SqlRequest request) in /tmp/gateway/Cosmos.Postgres.Core/Interop/SqlMessageFormatter.cs:line 767\n   at Microsoft.Azure.Cosmos.Postgres.Core.Interop.SqlMessageFormatter.FormatRequest(KestrelHttpRequestContext transportRequestContext) in /tmp/gateway/Cosmos.Postgres.Core/Interop/SqlMessageFormatter.cs:line 117\n   at Microsoft.Azure.Cosmos.Postgres.Core.Interop.SqlRequestPipeline.ProcessRequestAsync(KestrelHttpRequestContext transportRequest) in /tmp/gateway/Cosmos.Postgres.Core/Interop/SqlRequestPipeline.cs:line 46"
 }

@TheovanKraay
Copy link
Member

Currently data explorer in the new preview emulator is hosted on a different port (1234) and without the additional paths. Try http://localhost:1234, that should take you to data explorer. We didn't anticipate that not preserving the old layout would be a problem for anyone (though clearly we need to make this clearer in the docs). Let us know if thats not the case!

@UberAtziri
Copy link

Currently data explorer in the new preview emulator is hosted on a different port (1234) and without the additional paths. Try http://localhost:1234, that should take you to data explorer. We didn't anticipate that not preserving the old layout would be a problem for anyone (though clearly we need to make this clearer in the docs). Let us know if thats not the case!

Thanks for the quick response. I get ERR_EMPTY_RESPONSE when I try to open 1234 port in chrome

@TheovanKraay
Copy link
Member

Currently data explorer in the new preview emulator is hosted on a different port (1234) and without the additional paths. Try http://localhost:1234, that should take you to data explorer. We didn't anticipate that not preserving the old layout would be a problem for anyone (though clearly we need to make this clearer in the docs). Let us know if thats not the case!

Thanks for the quick response. I get ERR_EMPTY_RESPONSE when I try to open 1234 port in chrome

Ok. Would you mind opening a new issue so that we can keep this separate from the GA/older emulator? We might need to switch on telemetry and/or ask more questions to reproduce etc. Please tag the new issue with cosmosEmulatorVnextPreview.Thanks!

@mlabrum
Copy link

mlabrum commented Nov 21, 2024

Hi Team,

it's amazing that you've launched the new emulator; my team swapped away from using cosmos DB due to the existing emulator issues within CI/CD/local development. Can I ask what the support for this version of the emulator will be going forward?

If we choose to build new projects using this, will we be in the same situation as last time where it seemed Microsoft had abandoned this project or is there additional ongoing support being given to this project going forward?

@jonas-lomholdt
Copy link

Hello all - please try out the new linux-based emulator, it has been released today in preview, and will run on macOS M1 and other previously unsupported platforms - see readme right here in this repo. See also here for feature support. If you find issues or it doesn't support a feature that is important to you, please raise an issue in this repo, and tag it with cosmosEmulatorVnextPreview. Thanks!

It works! I'm so happy I could cry!

Tears of happiness! This is fantastic! Thanks 🚀

@TheovanKraay
Copy link
Member

Hi Team,

it's amazing that you've launched the new emulator; my team swapped away from using cosmos DB due to the existing emulator issues within CI/CD/local development. Can I ask what the support for this version of the emulator will be going forward?

If we choose to build new projects using this, will we be in the same situation as last time where it seemed Microsoft had abandoned this project or is there additional ongoing support being given to this project going forward?

Hello @mlabrum - development of the next generation emulator has full support from the Azure Cosmos DB engineering team here at Microsoft. As you can see, it is in limited preview and several features are not yet added. However, we are working towards a GA version, and at that point we expect this version of the emulator to largely replace the current stable version. This is an early preview to get feedback and help prioritise closing feature gaps.

@wondertalik
Copy link

Hey, folks. Thank you very much MS team for the emulator. We will move our team to use it for test purpose. I tried locally on my macbook m3 pro and looks like it works.

@vivaladan
Copy link

Really happy to see you release this, even in preview. Thankyou for the effort.

Has anyone gotten it working with Aspire yet? The container starts up, but then as Aspire see's it, it never becomes healthy.

@markjbrown
Copy link

Really happy to see you release this, even in preview. Thankyou for the effort.

Has anyone gotten it working with Aspire yet? The container starts up, but then as Aspire see's it, it never becomes healthy.

Here is a gist on how to do it. https://gist.github.com/christopheranderson/f75d8665306f418753c2010453648444

We'll have official docs on this soon.

Thanks.

@kmandalas
Copy link

kmandalas commented Nov 23, 2024

I tried using it at a Java/Spring Boot project. The container does start and connection is established. When I tried running my integration tests (using TestContainers) at the very first test I get the following error:

com.azure.cosmos.CosmosException: {"innerErrorMessage":"Index does not have options 16, StatusCode: InternalServerError"...}

I guess this is due to the unsupported features listed in https://learn.microsoft.com/en-us/azure/cosmos-db/emulator-linux#feature-support?

@markjbrown
Copy link

I tried using it at a Java/Spring Boot project. The container does start and connection is established. When I tried running my integration tests (using TestContainers) at the very first test I get the following error:

com.azure.cosmos.CosmosException: {"innerErrorMessage":"Index does not have options 16, StatusCode: InternalServerError"...}

I guess this is due to the unsupported features listed in https://learn.microsoft.com/en-us/azure/cosmos-db/emulator-linux#feature-support?

Hi @kmandalas can you raise a new issue and tag it with this? cosmosEmulatorVnextPreview. This will make it easier for us to track new questions/issues related to the preview version. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests