Adds an RFC to implement lineage backend

[verdan]

Signed-off-by: verdan [email protected]

[dorianj]

This is somewhat under-specified on its own (could specify the exact models etc), however I think given the prior RFC that went out, and existing code, that figuring out the details in implementation should be fine.

[dorianj]

It would be reasonable to define this in the RFC. That said, given there's already table lineage, and the metadata response formats are defined, I think the solution space is small enough it's probably OK to figure it out in the implementation

[NiravLangaliya]

I agree, because we have started to define neo4j queries based up on pervious RFC and also lineage model available into amundsen-common. though these are not optimized queries and not completed but I was planning to put queries on this line.

    def get_lineage(self, *,
                    id: str,
                    resource_type: ResourceType, direction: str, depth: int) -> Lineage:

        get_both_lineage_query = textwrap.dedent(u"""
        MATCH (down_parent:Table)<-[downstream_len:DOWNSTREAM*..{depth_key}]-(child:Table {{key: $query_key }})-[upstream_len:UPSTREAM*..{depth_key}]->(up_parent:Table)
        WITH
        child.key as child_key
        ,collect(distinct{{level:LENGTH(upstream_len),source:'hive',key:up_parent.key}}) AS upstream_entities
        ,collect(distinct{{level:LENGTH(downstream_len),source:'hive',key:down_parent.key}}) AS downstream_entities
        RETURN
        collect({{
        key:child_key,direction:"both",depth:1
        ,upstream_entities:upstream_entities
        ,downstream_entities:downstream_entities
        }}) AS lineageOutput
        """).format(depth_key=depth)

        get_upstream_lineage_query = textwrap.dedent(u"""
        MATCH (child:Table {{key: $query_key }})-[upstream_len:UPSTREAM*..{depth_key}]->(up_parent:Table)
        WITH
        child.key as child_key
        ,collect(distinct{{level:LENGTH(upstream_len),source:'hive',key:up_parent.key}}) AS upstream_entities
        RETURN
        collect({{
        key:child_key,direction:"upstream",depth:1
        ,upstream_entities:upstream_entities
        }}) AS lineageOutput
        """).format(depth_key=depth)

        get_downstream_lineage_query = textwrap.dedent(u"""
        MATCH (down_parent:Table)<-[downstream_len:DOWNSTREAM*..{depth_key}]-(child:Table {{key: $query_key }})
        WITH
        child.key as child_key
        ,collect(distinct{{level:LENGTH(downstream_len),source:'hive',key:down_parent.key}}) AS downstream_entities
        RETURN
        collect({{
        key:child_key,direction:"downstream",depth:1
        ,downstream_entities:downstream_entities
        }}) AS lineageOutput
        """).format(depth_key=depth)

        if direction == 'upstream':
            records = self._execute_cypher_query(statement=get_upstream_lineage_query,
                                                 param_dict={'query_key': id})
        elif direction == 'downstream':
            records = self._execute_cypher_query(statement=get_downstream_lineage_query,
                                                 param_dict={'query_key': id})
        else:
            records = self._execute_cypher_query(statement=get_both_lineage_query,
                                                 param_dict={'query_key': id})

        result = records.single()['lineageOutput'][0]
        return result

[feng-tao]

could you describe whether backend is only for graph or atlas or mysql? which one does it not plan to support?

[dorianj]

added

[feng-tao]

do you plan to change the existing model?
If so, what will be the new model interface?

The current one doesn't take into account for job/application that generate the lineage in between, do you plan to add those?

[dorianj]

clarified

[dorianj]

@feng-tao Added some clarifications to answer your questions, thanks for the feedback!

[dorianj]

added

[dorianj]

clarified