[Data Factory]Add ORC Dataset/Copy Source/Copy Sink #7040

TinaHu1 · 2019-08-26T01:52:35Z

Latest improvements:

MSFT employees can try out our new experience at OpenAPI Hub - one location for using our validation tools and finding your workflow.

Contribution checklist:

I have reviewed the documentation for the workflow.
Validation tools were run on swagger spec(s) and have all been fixed in this PR.
The OpenAPI Hub was used for checking validation status and next steps.

ARM API Review Checklist

Service team MUST add the "WaitForARMFeedback" label if the management plane API changes fall into one of the below categories.
adding/removing APIs.
adding/removing properties.
adding/removing API-version.
adding a new service in Azure.

Failure to comply may result in delays for manifest application. Note this does not apply to data plane APIs.

If you are blocked on ARM review and want to get the PR merged urgently, please get the ARM oncall for reviews (RP Manifest Approvers team under Azure Resource Manager service) from IcM and reach out to them.
Please follow the link to find more details on API review process.

Merge from azure master

merge from master

Merge from azure-master

openapi-sdkautomation · 2019-08-26T01:52:38Z

In Testing, Please Ignore

[Logs] (Generated from 864a2b3, Iteration 3)

.NET: test-repo-billy/azure-sdk-for-net [Logs] [Diff]

Microsoft.Azure.Management.DataFactory [Logs] [Instructions]
- Microsoft.Azure.Management.DataFactory.4.1.2.nupkg
- Microsoft.Azure.Management.DataFactory.4.1.2.snupkg

Python: test-repo-billy/azure-sdk-for-python [Logs] [Diff]

azure-mgmt-datafactory [Logs] [Instructions] [Generation PR]
- azure-mgmt-datafactory-0.7.0.zip
- azure_mgmt_datafactory-0.7.0-py2.py3-none-any.whl

Java: test-repo-billy/azure-sdk-for-java [Logs] [Diff]

datafactory/resource-manager/v2017_09_01_preview [Logs] [Generation PR]
datafactory/resource-manager/v2018_06_01 [Logs]

Go: test-repo-billy/azure-sdk-for-go [Logs] [Diff]

datafactory/mgmt/2018-06-01 [Logs] [Instructions] [Generation PR] [Integration PR]
- datafactory.mgmt.2018-06-01.zip
preview/datafactory/mgmt/2017-09-01-preview [Logs]
- preview.datafactory.mgmt.2017-09-01-preview.zip

JavaScript: test-repo-billy/azure-sdk-for-js [Logs] [Diff]

No packages generated.

AutorestCI · 2019-08-26T01:55:20Z

Automation for azure-sdk-for-python

A PR has been created for you based on this PR content.

Once this PR will be merged, content will be added to your service PR:
Azure/azure-sdk-for-python#7023

AutorestCI · 2019-08-26T01:59:43Z

Automation for azure-sdk-for-go

The initial PR has been merged into your service PR:
Azure/azure-sdk-for-go#5679

azuresdkci · 2019-08-26T02:02:29Z

Can one of the admins verify this patch?

...atafactory/resource-manager/Microsoft.DataFactory/stable/2018-06-01/entityTypes/Dataset.json

Juliehzl · 2019-08-27T15:50:46Z

...tafactory/resource-manager/Microsoft.DataFactory/stable/2018-06-01/entityTypes/Pipeline.json

@@ -776,6 +776,21 @@
        }
      }
    },
+    "OrcSource": {
+      "description": "A copy activity ORC source.",


One more thing is that where OrcSource and OrcSink are used in your swagger file.

It is the similiar usage.
Take Source as an example:
In specification/datafactory/resource-manager/Microsoft.DataFactory/stable/2018-06-01/entityTypes/Pipeline.json
"OrcSource" is to add a new type for "CopySource" with the defination "discriminator" in copy source(line 914). Then "CopySource" is referenced by "CopyActivityTypeProperties"(line 885)=>"CopyActivity"(line 859)=>extend a type for "Activity"(line 21)=>"Pipeline"(line 9)

In specification/datafactory/resource-manager/Microsoft.DataFactory/stable/2018-06-01/datafactory.json
"Pipeline" was reference by "PipelineResource"(line 4106)=>used in several API.

Sink is almost the same as Source.

Why do you use x-ms-discriminator-value in "OrcDataset" not in "OrcSource"?

Because the real value we want for customer to use is "Orc" but not "OrcDataset".
@Juliehzl It will be very helpful if we can merge this PR before tomorrow, thanks.

@TinaHu1 Thanks for your detailed explanation. Just ensure everything is in the right way. 😊

...atafactory/resource-manager/Microsoft.DataFactory/stable/2018-06-01/entityTypes/Dataset.json

ravbhatnagar · 2019-08-29T16:59:41Z

yes, @anthony-c-martin we have discussed this in the past. And although this is a breaking change, ADF scenario is bit more nuanced. So, we have let this through in the past with the expectation that we will bring this up with MarkRus and other folks and seek guidance for such scenarios.
So for now, this should be ok to sign off.

anthony-c-martin

As per Gaurav's comment, signing off even though this is a breaking change.

TinaHu1 · 2019-08-30T05:38:06Z

@Juliehzl Could you help merge this PR? Thanks

Juliehzl · 2019-08-31T02:06:10Z

@TinaHu1 I am waiting for your answer. With your clear explanation, I can understand "OrcDataset" now. I wonder why OrcSource and OrcSink are not used but you defined.

TinaHu1 and others added 4 commits August 9, 2019 10:20

Merge pull request #1 from Azure/master

de493fa

Merge from azure master

Merge pull request #3 from Azure/master

07dcf30

merge from master

Merge pull request #4 from Azure/master

b90d8e6

Merge from azure-master

[DataFactory]Add support for ORC dataset

864a2b3

AutorestCI mentioned this pull request Aug 26, 2019

[AutoPR datafactory/resource-manager] [Data Factory]Add ORC Dataset/Copy Source/Copy Sink Azure/azure-sdk-for-python#6935

Closed

AutorestCI mentioned this pull request Aug 26, 2019

[AutoPR datafactory/resource-manager] [Data Factory]Add ORC Dataset/Copy Source/Copy Sink Azure/azure-sdk-for-go#5611

Merged

openapi-sdkautomation bot mentioned this pull request Aug 26, 2019

[AutoPR Microsoft.Azure.Management.DataFactory] [Data Factory]Add ORC Dataset/Copy Source/Copy Sink test-repo-billy/azure-sdk-for-net#143

Merged

azuresdkci assigned Juliehzl Aug 26, 2019

azuresdkci requested a review from Juliehzl August 26, 2019 02:15

Juliehzl reviewed Aug 26, 2019

View reviewed changes

...atafactory/resource-manager/Microsoft.DataFactory/stable/2018-06-01/entityTypes/Dataset.json Show resolved Hide resolved

openapi-sdkautomation bot mentioned this pull request Aug 27, 2019

[AutoPR datafactory/resource-manager/v2017_09_01_preview] [Data Factory]Add ORC Dataset/Copy Source/Copy Sink test-repo-billy/azure-sdk-for-java#262

Closed

Juliehzl added the WaitForARMFeedback <valid label in PR review process> add this label when ARM review is required label Aug 27, 2019

Juliehzl reviewed Aug 27, 2019

View reviewed changes

anthony-c-martin requested changes Aug 28, 2019

View reviewed changes

...atafactory/resource-manager/Microsoft.DataFactory/stable/2018-06-01/entityTypes/Dataset.json Show resolved Hide resolved

anthony-c-martin added ARMSignedOff <valid label in PR review process>add this label when ARM approve updates after review and removed WaitForARMFeedback <valid label in PR review process> add this label when ARM review is required labels Aug 29, 2019

anthony-c-martin self-requested a review August 29, 2019 17:02

anthony-c-martin approved these changes Aug 29, 2019

View reviewed changes

Juliehzl approved these changes Sep 3, 2019

View reviewed changes

Juliehzl merged commit 935dfad into Azure:master Sep 3, 2019

TinaHu1 mentioned this pull request Oct 11, 2019

[DataFactory]Add CompressionCodec property for ORC Dataset #7452

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Data Factory]Add ORC Dataset/Copy Source/Copy Sink #7040

[Data Factory]Add ORC Dataset/Copy Source/Copy Sink #7040

TinaHu1 commented Aug 26, 2019 •

edited

Loading

openapi-sdkautomation bot commented Aug 26, 2019 •

edited

Loading

AutorestCI commented Aug 26, 2019 •

edited

Loading

AutorestCI commented Aug 26, 2019 •

edited

Loading

azuresdkci commented Aug 26, 2019

Juliehzl Aug 27, 2019

TinaHu1 Sep 1, 2019

Juliehzl Sep 2, 2019

TinaHu1 Sep 2, 2019

Juliehzl Sep 3, 2019

ravbhatnagar commented Aug 29, 2019

anthony-c-martin left a comment

TinaHu1 commented Aug 30, 2019

Juliehzl commented Aug 31, 2019

[Data Factory]Add ORC Dataset/Copy Source/Copy Sink #7040

[Data Factory]Add ORC Dataset/Copy Source/Copy Sink #7040

Conversation

TinaHu1 commented Aug 26, 2019 • edited Loading

Latest improvements:

Contribution checklist:

ARM API Review Checklist

openapi-sdkautomation bot commented Aug 26, 2019 • edited Loading

In Testing, Please Ignore

[Logs] (Generated from 864a2b3, Iteration 3)

AutorestCI commented Aug 26, 2019 • edited Loading

Automation for azure-sdk-for-python

AutorestCI commented Aug 26, 2019 • edited Loading

Automation for azure-sdk-for-go

azuresdkci commented Aug 26, 2019

Juliehzl Aug 27, 2019

Choose a reason for hiding this comment

TinaHu1 Sep 1, 2019

Choose a reason for hiding this comment

Juliehzl Sep 2, 2019

Choose a reason for hiding this comment

TinaHu1 Sep 2, 2019

Choose a reason for hiding this comment

Juliehzl Sep 3, 2019

Choose a reason for hiding this comment

ravbhatnagar commented Aug 29, 2019

anthony-c-martin left a comment

Choose a reason for hiding this comment

TinaHu1 commented Aug 30, 2019

Juliehzl commented Aug 31, 2019

TinaHu1 commented Aug 26, 2019 •

edited

Loading

openapi-sdkautomation bot commented Aug 26, 2019 •

edited

Loading

AutorestCI commented Aug 26, 2019 •

edited

Loading

AutorestCI commented Aug 26, 2019 •

edited

Loading