META - Support for Windows and MacOS (local)

[trallard]

Background

Currently, conda-store is limited to Linux-based systems use.
To better serve our community, we'd need to ensure it is Windows and MacOS compatible.

Tasks

General

Preview Give feedback

1. [ENH] - Consider using OS-specific standards for conda-store data #648
   area: configuration ⚙️ area: user experience 👩🏻‍💻 project: challenges type: enhancement 💅🏼 +4
   

Windows

Preview Give feedback

1. Windows support #640
   +0
2. ENH - Shorten hash used in environment path #611
   needs: discussion 💬 type: enhancement 💅🏼 +2
   
3. Figure out how to handle long paths on Windows #588
   area: configuration ⚙️ area: user experience 👩🏻‍💻 project: challenges type: enhancement 💅🏼 +4
   

macOS

Preview Give feedback

1. Make the disk_usage() function work on macOS #537
   area: api 🌐 status: in progress 🏗 +2
2. Add a macOS worker to the CI #550
   area: testing ✅ needs: review 👀 status: in progress 🏗 +3
   

[costrouc]

@asmeurer releated to #513. And should be included with that work.

[asmeurer]

I was able to run conda-store-server with --standalone on macos. You have to comment out conda-docker from the environment-dev.yml and add python-docker to build the development environment.

When I tried creating an environment, I get

Traceback (most recent call last):
  File "/Users/aaronmeurer/Documents/conda-store/conda-store-server/conda_store_server/build.py", line 278, in build_conda_environment
    fetch_and_extract_packages(conda_store, pathlib.Path(tmp_lock_filename))
  File "/Users/aaronmeurer/Documents/conda-store/conda-store-server/conda_store_server/build.py", line 197, in fetch_and_extract_packages
    with filelock.FileLock(f"{str(filepath)}.lock"):
  File "/Users/aaronmeurer/anaconda3/envs/conda-store-server-dev/lib/python3.10/site-packages/filelock/_api.py", line 255, in __enter__
    self.acquire()
  File "/Users/aaronmeurer/anaconda3/envs/conda-store-server-dev/lib/python3.10/site-packages/filelock/_api.py", line 213, in acquire
    self._acquire()
  File "/Users/aaronmeurer/anaconda3/envs/conda-store-server-dev/lib/python3.10/site-packages/filelock/_unix.py", line 37, in _acquire
    fd = os.open(self.lock_file, open_flags, self._context.mode)
FileNotFoundError: [Errno 2] No such file or directory: '/Users/aaronmeurer/anaconda3/envs/conda-store-server-dev/pkgs/bzip2-1.0.8-h0d85af4_4.tar.bz2.lock'

in the build log. Note that I am building off of v0.4.15.2 because main is currently broken because of #520

[asmeurer]

It looks like this error is coming from the fact that the conda-store-server-dev environment has conda itself installed in it. This causes conda to think that it is the base environment and should have pkgs in ~/anaconda/envs/conda-store-server-dev/pkgs. I think on my Linux machine I ended up using conda to install something else while the environment was active and it created this directory with a bunch of packages, which is why things worked for me there. But on my Mac I never did that and the pkgs directory there doesn't exist.

I used CONDARC=condarc conda-store-server --standalone, where condarc is a file with

pkgs_dirs: ['/Users/aaronmeurer/anaconda3/pkgs']

and I was able to successfully create an environment.

Questions:

• is installing conda in an environment actually supported these days?
• If it is, is there a better way to do it so that it uses the original package cache?

This smells to me like conda-store-server is actually doing something wrong here. It shouldn't fail because some files in the pkgs cache fail to exist. It's likely just a latent bug here that never gets triggered because the pkgs cache always does end up existing. I'm a little curious what ends up happening here inside of the Docker container build.

[asmeurer]

Anyway, I was able to successfully create an environment (as in the environment builds and I can import the packages installed in it).

I now get this error when building an environment:

du: invalid option -- b
usage: du [-Aclnx] [-H | -L | -P] [-g | -h | -k | -m] [-a | -s | -d depth] [-B blocksize] [-I mask] [-t threshold] [file ...]
[2023-08-15 15:53:34,557: WARNING/ForkPoolWorker-10] [CondaStoreWorker] ERROR | Command '['du', '-sb', '/Users/aaronmeurer/Documents/conda-store/conda-store-server/conda-store-state/default/fa21557fc509dce04a1d2c7671d4c879491ae5885924f03f167d64cc5b9402e5-20230815-215226-377063-4-test4']' returned non-zero exit status 64.
Traceback (most recent call last):
  File "/Users/aaronmeurer/Documents/conda-store/conda-store-server/conda_store_server/build.py", line 201, in build_conda_environment
    context = action.action_get_conda_prefix_stats(conda_prefix)
  File "/Users/aaronmeurer/Documents/conda-store/conda-store-server/conda_store_server/action/base.py", line 31, in wrapper
    action_context.result = f(action_context, *args, **kwargs)
  File "/Users/aaronmeurer/Documents/conda-store/conda-store-server/conda_store_server/action/get_conda_prefix_stats.py", line 13, in action_get_conda_prefix_stats
    stats["disk_usage"] = int(utils.disk_usage(conda_prefix))
  File "/Users/aaronmeurer/Documents/conda-store/conda-store-server/conda_store_server/utils.py", line 53, in disk_usage
    return subprocess.check_output(["du", "-sb", str(path)], encoding="utf-8").split()[
  File "/Users/aaronmeurer/anaconda3/envs/conda-store-server-dev/lib/python3.10/subprocess.py", line 420, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/Users/aaronmeurer/anaconda3/envs/conda-store-server-dev/lib/python3.10/site-packages/conda_lock/_vendor/poetry/utils/_compat.py", line 217, in run
    raise CalledProcessError(
conda_lock._vendor.poetry.utils._compat.CalledProcessError: Command '['du', '-sb', '/Users/aaronmeurer/Documents/conda-store/conda-store-server/conda-store-state/default/fa21557fc509dce04a1d2c7671d4c879491ae5885924f03f167d64cc5b9402e5-20230815-215226-377063-4-test4']' returned non-zero exit status 64.

which looks pretty easy to fix.

[asmeurer]

Fix for the du thing at #537. I now get issues related to conda-docker, which is a known problem (conda-docker doesn't work on Mac and so docker building should just be disabled when we aren't on linux for now).

[costrouc]

@asmeurer totally agree that we need to adjust the defaults and make the conda docker build not default on non-linux systems.

A few places where this needs to be configured:

• conda-store/conda-store-server/conda_store_server/app.py

  Lines 238 to 248 in b6be123

  	build_artifacts = List(
  	[
  	schema.BuildArtifactType.LOCKFILE,
  	schema.BuildArtifactType.YAML,
  	schema.BuildArtifactType.CONDA_PACK,
  	schema.BuildArtifactType.DOCKER_MANIFEST,
  	schema.BuildArtifactType.CONTAINER_REGISTRY,
  	],
  	help="artifacts to build in conda-store. By default all of the artifacts",
  	config=True,
  	)

• conda-store/conda-store-server/conda_store_server/schema.py

  Line 318 in b6be123

  build_artifacts: List[BuildArtifactType] = Field(

I'm guessing when platform is not linux BuildArtifactType.DOCKER_MANIFEST. Should not be included.

For the issues releated to conda-docker imports. I'd be fine if we just more the conda_docker imports into action function itself so on windows/osx this module isn't required

conda-store/conda-store-server/conda_store_server/action/generate_conda_docker.py

Lines 3 to 9 in b6be123

	from conda_docker.conda import (
	build_docker_environment_image,
	conda_info,
	fetch_precs,
	find_user_conda,
	precs_from_environment_prefix,
	)

[costrouc]

@asmeurer thanks for checking and getting it to the point that OSX is working!

Two things I think that are next:

• adding tests to the CI that we could run just to ensure this functionality is working using osx runners
  • this should spin up the conda-store-server --standalone and run some basic tests we don't need much since the actual functionality is being testing in other tests.
• working on ensuring that windows is supported (since this hasn't been addressed yet). I would think this is a higher priority for JATIC

[asmeurer]

I'm pushing my fixes for Windows to #559

The first thing I notice on Windows is that the "set conda prefix permissions" action is tied to Unix permissions. How do we want to handle this on Windows?

Note that currently conda_store_server cannot even be imported on Windows because this line fails

conda-store/conda-store-server/conda_store_server/app.py

Line 305 in f62255c

os.getuid(),

[trallard]

Tagging @costrouc for insights

[asmeurer]

A much bigger problem: celery doesn't officially support Windows https://docs.celeryq.dev/en/stable/faq.html#windows. From what I've read, it is often possible to get it to work on Windows anyways (see https://stackoverflow.com/questions/37255548/how-to-run-celery-on-windows and https://distributedpython.com/posts/two-ways-to-make-celery-4-run-on-windows/).

It's not clear yet to me how much work this would be and how reliable it would be if we did get it working. Already, if you try to start the worker process on Windows, you get an error

PS C:\Users\Aaron\Quansight\conda-store\conda-store-server> C:\Users\Aaron\miniconda3\envs\conda-store-server-dev\python.exe -m conda_store_server.worker
Traceback (most recent call last):
  File "C:\Users\Aaron\miniconda3\envs\conda-store-server-dev\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\Aaron\miniconda3\envs\conda-store-server-dev\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\Aaron\Quansight\conda-store\conda-store-server\conda_store_server\worker\__main__.py", line 6, in <module>
    main()
  File "C:\Users\Aaron\miniconda3\envs\conda-store-server-dev\lib\site-packages\traitlets\config\application.py", line 1043, in launch_instance
    app.start()
  File "C:\Users\Aaron\Quansight\conda-store\conda-store-server\conda_store_server\worker\app.py", line 82, in start
    self.conda_store.celery_app.worker_main(argv)
  File "C:\Users\Aaron\miniconda3\envs\conda-store-server-dev\lib\site-packages\celery\app\base.py", line 388, in worker_main
    self.start(argv=argv)
  File "C:\Users\Aaron\miniconda3\envs\conda-store-server-dev\lib\site-packages\celery\app\base.py", line 368, in start
    celery.main(args=argv, standalone_mode=False)
  File "C:\Users\Aaron\miniconda3\envs\conda-store-server-dev\lib\site-packages\click\core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "C:\Users\Aaron\miniconda3\envs\conda-store-server-dev\lib\site-packages\click\core.py", line 1686, in invoke
    sub_ctx = cmd.make_context(cmd_name, args, parent=ctx)
  File "C:\Users\Aaron\miniconda3\envs\conda-store-server-dev\lib\site-packages\click\core.py", line 943, in make_context
    self.parse_args(ctx, args)
  File "C:\Users\Aaron\miniconda3\envs\conda-store-server-dev\lib\site-packages\click\core.py", line 1408, in parse_args
    value, args = param.handle_parse_result(ctx, opts, args)
  File "C:\Users\Aaron\miniconda3\envs\conda-store-server-dev\lib\site-packages\click\core.py", line 2400, in handle_parse_result
    value = self.process_value(ctx, value)
  File "C:\Users\Aaron\miniconda3\envs\conda-store-server-dev\lib\site-packages\click\core.py", line 2356, in process_value
  File "C:\Users\Aaron\miniconda3\envs\conda-store-server-dev\lib\site-packages\click\core.py", line 2344, in type_cast_value
    return convert(value)
  File "C:\Users\Aaron\miniconda3\envs\conda-store-server-dev\lib\site-packages\click\core.py", line 2316, in convert
    return self.type(value, param=self, ctx=ctx)
  File "C:\Users\Aaron\miniconda3\envs\conda-store-server-dev\lib\site-packages\click\types.py", line 83, in __call__
    return self.convert(value, param, ctx)
  File "C:\Users\Aaron\miniconda3\envs\conda-store-server-dev\lib\site-packages\celery\bin\worker.py", line 29, in convert
    self.fail('-B option does not work on Windows.  '
  File "C:\Users\Aaron\miniconda3\envs\conda-store-server-dev\lib\site-packages\click\types.py", line 136, in fail
    raise BadParameter(message, ctx=ctx, param=param)
click.exceptions.BadParameter: -B option does not work on Windows.  Please run celery beat as a separate service.

[pavithraes]

From today's meeting - @asmeurer can spend some time looking into workarounds (or alternatives) for celery / windows support

[asmeurer]

Using some of the workarounds suggested at https://stackoverflow.com/questions/37255548/how-to-run-celery-on-windows, I am able to get celery working enough where I can at least start fixing the issues with individual actions on Windows (there seem to be quite a few). It's a little hard to tell with things still in a nonworking state, but things appear to be quite slow, presumably because the workarounds disable parallelism.

[kcpevey]

Waiting on PR review from @costrouc

[nkaretnikov]

Initial Windows support has just landed as part of #640 (most of the work there was done by Aaron). It makes it possible to run on Windows in --standalone mode, with SQLite as the DB. A known limitation is Windows path length: users are expected to adjust that via the registry for now. It's "initial support" because only basic manual testing was done + CI. Celery is not officially supported on Windows, too. So there might be bugs. Please report an issue if you see any.

[costrouc]

This issue has already been addressed in #640. @nkaretnikov needs to go through this issue and make sure that all the subtasks have been addressed.

[nkaretnikov]

Potential issues here:

• celery is not officially supported on Windows (need to try parallel builds, this can be checked with an API test, need to make sure multiple workers are running) [ENH] - Test for parallel builds #548
• du calculation that we implemented in Python might be slightly wrong on Macs (IIRC it has the biggest error of all three platforms)
• See Revisit the implementation of --standalone as well as the assumptions made #513 (comment).

[asmeurer]

Also to be noted:

• The --beat flag to celery is disabled on Windows (celery doesn't support it there). I don't know what implications this has.
• As on Mac, building Docker containers isn't supported on Windows.
• Filesystem permissions stuff is completely ignored on Windows, because it currently is built around Unix permissions. It's probably best to just leave this until someone requests it.