Agent Metadata Servicer

[Future-Outlier]

Tracking issue

flyteorg/flyte#4511

Describe your changes

Implement Agent Metadata Servicer in the agent server.

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Setup Process

tasks:
  task-plugins:
    enabled-plugins:
      - container
      - sidecar
      - K8S-ARRAY
      - agent-service
  default-for-task-types:
      api_task: agent-service
      airflow: agent-service
      sensor: agent-service
      spark: agent-service
      bigquery_query_job_task: agent-service
      custom_task: agent-service
      container: container
      container_array: K8S-ARRAY
agent-service:
   defaultAgent:
      endpoint: "dns:///localhost:8000"
      insecure: true
      timeouts:
        GetTask: 100s
      defaultTimeout: 100s

1. start the single binary mode

flytectl demo start --dev

2. start the agent server

pyflyte serve agent

3. start the flyte cluster

flyte start --config ./flyte-single-binary-local-dev.yaml

Screenshots

{
   "json":{
      "src":"plugin.go:391"
   },
   "level":"info",
   "msg":"Agent supports task types: [sensor spark airflow task_type_1 task_type_2]",
   "ts":"2023-12-19T22:23:03+08:00"
}

Note to reviewers

Related PRs

[codecov]

Codecov Report

Attention: 8 lines in your changes are missing coverage. Please review.

Comparison is base (6279b81) 85.79% compared to head (e7afa5d) 85.79%.
Report is 26 commits behind head on master.

Files	Patch %	Lines
flytekit/extend/backend/base_agent.py	64.28%	3 Missing and 2 partials ⚠️
flytekit/extend/backend/agent_service.py	57.14%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2012      +/-   ##
==========================================
- Coverage   85.79%   85.79%   -0.01%     
==========================================
  Files         313      313              
  Lines       23507    23618     +111     
  Branches     3514     3539      +25     
==========================================
+ Hits        20168    20262      +94     
- Misses       2730     2744      +14     
- Partials      609      612       +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.