Creates secret_manager module and adds read_secret

[desertaxle]

No description provided.

[ahuang11]

Would it work with just: raise ValueError(f"Unable to get value for secret {secret_name}")

[desertaxle]

Yes, but this is a good opportunity to talk about how we want to handle exceptions in shared tasks. I generally will allow exceptions to bubble up because they will contain more information about how to resolve the error and we don't have to code for all possible error cases. What are your thoughts?

[ahuang11]

Doesn't raise still create a full traceback?

def test():
    try:
        1 / 0
    except Exception:
        print("Failed")
        raise

test()

results in

---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)
Input In [6], in <module>
      5         print("Failed")
      6         raise
----> 8 test()

Input In [6], in test()
      1 def test():
      2     try:
----> 3         1 / 0
      4     except Exception:
      5         print("Failed")

ZeroDivisionError: division by zero

vs

def test():
    try:
        1 / 0
    except Exception:
        raise ZeroDivisionError("Failed")

test()

results in

---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)
Input In [7], in test()
      2 try:
----> 3     1 / 0
      4 except Exception:

ZeroDivisionError: division by zero

During handling of the above exception, another exception occurred:

ZeroDivisionError                         Traceback (most recent call last)
Input In [7], in <module>
      4     except Exception:
      5         raise ZeroDivisionError("Failed")
----> 7 test()

Input In [7], in test()
      3     1 / 0
      4 except Exception:
----> 5     raise ZeroDivisionError("Failed")

ZeroDivisionError: Failed

[ahuang11]

Although maybe I should try with orion + logger

[desertaxle]

I prefer the first traceback that you posted. I think that raising another exception during error handling tends to obfuscate the original exception.

[ahuang11]

The original version:

from prefect import flow
from prefect import get_run_logger

@flow
def test():
    logger = get_run_logger()
    try:
        1 / 0
    except Exception:
        logger.exception("Failed")
        raise

test()

11:18:50.838 | INFO    | prefect.engine - Created flow run 'vigorous-kestrel' for flow 'test'
11:18:50.839 | INFO    | Flow run 'vigorous-kestrel' - Using task runner 'ConcurrentTaskRunner'
11:18:50.853 | ERROR   | Flow run 'vigorous-kestrel' - Failed
Traceback (most recent call last):
  File "/var/folders/tb/fksvh8ms2hl5xnzwrftnx66r0000gq/T/ipykernel_59130/4129042268.py", line 8, in test
    1 / 0
ZeroDivisionError: division by zero
11:18:50.854 | ERROR   | Flow run 'vigorous-kestrel' - Encountered exception during execution:
Traceback (most recent call last):
  File "/Users/andrew/orion/src/prefect/engine.py", line 445, in orchestrate_flow_run
    result = await run_sync_in_worker_thread(flow_call)
  File "/Users/andrew/orion/src/prefect/utilities/asyncio.py", line 51, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(context.run, call, cancellable=True)
  File "/Users/andrew/mambaforge/envs/azure/lib/python3.9/site-packages/anyio/to_thread.py", line 28, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(func, *args, cancellable=cancellable,
  File "/Users/andrew/mambaforge/envs/azure/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 818, in run_sync_in_worker_thread
    return await future
  File "/Users/andrew/mambaforge/envs/azure/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 754, in run
    result = context.run(func, *args)
  File "/var/folders/tb/fksvh8ms2hl5xnzwrftnx66r0000gq/T/ipykernel_59130/4129042268.py", line 8, in test
    1 / 0
ZeroDivisionError: division by zero
11:18:50.872 | ERROR   | Flow run 'vigorous-kestrel' - Finished in state Failed('Flow run encountered an exception.')
Failed(message='Flow run encountered an exception.', type=FAILED, result=ZeroDivisionError('division by zero'), flow_run_id=3af5dbf4-0dea-4221-a513-985b4bcf48e9)

[ahuang11]

The raise version with {e}

@flow
def test():
    logger = get_run_logger()
    try:
        1 / 0
    except Exception as e:
        raise ZeroDivisionError(f"<ADDITIONAL HELPER INFO>: {e}")

test()

11:18:32.612 | INFO    | prefect.engine - Created flow run 'primitive-jerboa' for flow 'test'
11:18:32.612 | INFO    | Flow run 'primitive-jerboa' - Using task runner 'ConcurrentTaskRunner'
11:18:32.629 | ERROR   | Flow run 'primitive-jerboa' - Encountered exception during execution:
Traceback (most recent call last):
  File "/var/folders/tb/fksvh8ms2hl5xnzwrftnx66r0000gq/T/ipykernel_59130/2472474932.py", line 5, in test
    1 / 0
ZeroDivisionError: division by zero

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/andrew/orion/src/prefect/engine.py", line 445, in orchestrate_flow_run
    result = await run_sync_in_worker_thread(flow_call)
  File "/Users/andrew/orion/src/prefect/utilities/asyncio.py", line 51, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(context.run, call, cancellable=True)
  File "/Users/andrew/mambaforge/envs/azure/lib/python3.9/site-packages/anyio/to_thread.py", line 28, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(func, *args, cancellable=cancellable,
  File "/Users/andrew/mambaforge/envs/azure/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 818, in run_sync_in_worker_thread
    return await future
  File "/Users/andrew/mambaforge/envs/azure/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 754, in run
    result = context.run(func, *args)
  File "/var/folders/tb/fksvh8ms2hl5xnzwrftnx66r0000gq/T/ipykernel_59130/2472474932.py", line 7, in test
    raise ZeroDivisionError(f"<ADDITIONAL HELPER INFO>: {e}")
ZeroDivisionError: <ADDITIONAL HELPER INFO>: division by zero
11:18:32.646 | ERROR   | Flow run 'primitive-jerboa' - Finished in state Failed('Flow run encountered an exception.')
Failed(message='Flow run encountered an exception.', type=FAILED, result=ZeroDivisionError('<ADDITIONAL HELPER INFO>: division by zero'), flow_run_id=d247c848-4551-4339-9e69-9962236480b5)

[desertaxle]

Which of those do you prefer? Log + raise or raise new error?

[ahuang11]

Log + raise seems cleaner.

[ahuang11]

aws_credentials: AwsCredentials needs documentation

[desertaxle]

aws_credentials: AwsCredentials needs documentation

It's documented now!