Add functionality for .flyteignore file

[dansola]

Tracking issue

Linear Ticket: https://linear.app/unionai/issue/COR-1133/add-flyteignore-functionality-to-pyflyte-register

Why are the changes needed?

Sometimes users have files they want to ignore in the registration process that do not suit a dockerignore or gitignore.

What changes were proposed in this pull request?

Add a FlyteIgnore class that looks for a .flyteignore class in the root directory and ignores any files in it when computing a hash.

How was this patch tested?

• Unit tests.
• Local scenario with symlink:

.
├── lp.py
├── original_link.txt -> /Users/danielsola/repos/wine-classification/original.txt
└── wf.py

lp.py is

from flytekit import LaunchPlan
from wf import hello_world_wf

LaunchPlan.get_or_create(
    hello_world_wf,
    name="hello_lp",
)

and wf.py is

from flytekit import task, workflow

@task
def say_hello() -> str:
    return "Hello, World!"

@workflow
def hello_world_wf() -> str:
    res = say_hello()
    return res

Then I create a random file like this:
echo "This is the original file." > ~/path/to/repo/original.txt
and the symlink in the directory I showed you like this:
ln -s ~/repos/wine-classification/original.txt ~/path/to/repo/register_example/original_link.txt
Then I move the original file like this:
mv ~/path/to/repo/original.txt ~/path/to/repo/original-moved.txt
and then run
pyflyte register lp.py

With this I get an error
Failed with Unknown Exception <class 'FileNotFoundError'> Reason: [Errno 2] No such file or directory: '/path/to/repo/register_example/original_link.txt'

Then I add *.txt to a .flyteignore and I no longer get a failure.

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

Original PR for gitignore and dockerignore: #899

Docs link

[wild-endeavor]

we may end up removing the dependency in the future, but while we have it, why not just use the PatternMatcher object that the dockerignore object uses?

[dansola]

haha just pushed that! since my test was actually failing

[wild-endeavor]

This looks good to merge, and this is not the first request we've heard about having a separate flyteignore file, but a couple things

• this seems like the wrong solution to the particular issue mentioned above. The real bug is in the mechanism in flytekit that was looking for a symlinked file that no longer exists. That should be fixed in a separate PR (@dansola we can make another GH issue for that later).
• This is pretty naive ignoring and it may be worth it to mention that somewhere in a comment - proper git ignores support nested gitignore files, reverse ignoring, etc.

[welcome]

Congrats on merging your first pull request! 🎉