[Feature Request]: Build beam.MLTransform

[AnandInguva]

What would you like to happen?

We aim to create an easy-to-use PTransform in Apache Beam, called MLTransform, for carrying out common machine learning transforms on large datasets. MLTransform is designed to be framework-agnostic, and its primary goal is to provide an intuitive interface for users to perform various data processing transformations without writing complex code or dealing with underlying libraries. The first framework that we will make use of is TensorFlow Transform (TFT) which has many production hardened transformations already written using the Apache Beam primitives.

Design doc: https://docs.google.com/document/d/1rQkSm_8tseLqDQaLohtlCGqt5pvMaP0XIpPi5UD0LCQ/edit#
dev list discussion: https://lists.apache.org/thread/d4thp7xs1y0jm5m9v5xzshln9fwvsm7s

Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[AnandInguva]

v0 PR: #26795

[AnandInguva]

Work to do:

1. Add support for pa.RecordBatch - ref: MLTransform #26795 (comment)

[brucearctor]

@AnandInguva should this issue be closed?