-
Notifications
You must be signed in to change notification settings - Fork 5.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
BREAKING CHANGE: Adl object refactor #1429
Conversation
This does the following: * Create an explicit set of parameters for job submission. NOTE: Only U-SQL is supported in this change, no hive jobs are supported at present. * Create a "Basic" jobInformation that is returned for LIST calls * Setup inheritance for GET jobs * Create an inheritance structure for GET and LIST ADLS and ADLA accounts. * This also follows the Basic<Object> -> Object inheritance pattern.
@veronicagg this PR should address the validation issues that were opened previously. Can you take a look and help confirm? Is there something I need to run to ensure this fixes those issues? |
@olydis is looking into why CI is failing here. It looks like it isn't properly running against Node 8 instead of Node 6. I will restart CI once it is fixed. |
So it seems that Travis always picks up the config from the very branch you're trying to merge, even if it wasn't touched. I.e. it doesn't perform a git merge before getting that file... I've rebased your PR (#1433) and it works, so we can take it from there. |
@begoldsm Model validation errors: https://travis-ci.org/Azure/azure-rest-api-specs/jobs/254655271 |
… into adl_object_refactor
* Missed some of the example changes * Missed Job_Build, which should have its own parameters object as well.
No modification for AutorestCI/azure-sdk-for-ruby |
No modification for AutorestCI/azure-sdk-for-node |
* There is no functional change in this commit * This change is purely for refactoring related to Ben's refactoring changes (which has already been accepted): Azure#1429 * Reordering fields in the example files to match with Ben's refactoring changes * Other minor changes or moving code around
* This change is mapped to this PR: Azure/azure-rest-api-specs#1429 * This commit is for ADLA
* This change is mapped to this PR: Azure/azure-rest-api-specs#1429 * This commit for ADLS
* This change is mapped to this PR: Azure/azure-rest-api-specs#1429
* This change is mapped to this PR: Azure/azure-rest-api-specs#1429 * Checking in the successful tests
* Code gen for ADLA * This change is mapped to this PR: Azure/azure-rest-api-specs#1429 * This commit is for ADLA * Code gen for ADLS * This change is mapped to this PR: Azure/azure-rest-api-specs#1429 * This commit for ADLS * Code gen for ADLS (part 2) * I missed this file! * Code gen for ADL (part 3) * Updating to the correct commit number: Azure/azure-rest-api-specs#1452 * Update Azure Batch to 5.1.0 (#3513) * Fixing publishing of symbols. Fixing issue# 3518 (#3519) * Fix Batch changelog references (#3517) * Do hydra to swagger specs migration for azure-sdk-for-net (#3482) * Do hydra to swagger specs migration for azure-sdk-for-net * Fix build failed problem * update the assemblyinfo file with the updated version 2.6.0 * Revert "update the assemblyinfo file with the updated version 2.6.0" This reverts commit a33beb0. * Updata the AssemblyFileVersion to 2.6.0.0 * revert package version * Make project files concise. * Add Provider test case * update the pr with new tests with session records * remove extra code * Fixing changelog.md * Fixing up changelog.md
* Code gen for ADLA * This change is mapped to this PR: Azure/azure-rest-api-specs#1429 * This commit is for ADLA * Code gen for ADLS * This change is mapped to this PR: Azure/azure-rest-api-specs#1429 * This commit for ADLS * Code gen for ADLS (part 2) * I missed this file! * Code gen for ADL (part 3) * Updating to the correct commit number: Azure/azure-rest-api-specs#1452 * Update Azure Batch to 5.1.0 (#3513) * Fixing publishing of symbols. Fixing issue# 3518 (#3519) * Fix Batch changelog references (#3517) * Do hydra to swagger specs migration for azure-sdk-for-net (#3482) * Do hydra to swagger specs migration for azure-sdk-for-net * Fix build failed problem * update the assemblyinfo file with the updated version 2.6.0 * Revert "update the assemblyinfo file with the updated version 2.6.0" This reverts commit a33beb0. * Updata the AssemblyFileVersion to 2.6.0.0 * revert package version * Make project files concise. * Add Provider test case * update the pr with new tests with session records * remove extra code * Fixing changelog.md * Fixing up changelog.md * [DataLake Analytics] Fixing changelog.md More descriptive and fixing indent issue that was not addressed in the previous commit * [DataLake Store] Fixing changelog.md More descriptive and fixing indent issue not addressed in a previous commit * [DataLake Analytics] Fixing changelog.md Adding an important note about retrieving a list of jobs and accounts
* Code gen for ADLA * This change is mapped to this PR: Azure/azure-rest-api-specs#1429 * This commit is for ADLA * Code gen for ADLS * This change is mapped to this PR: Azure/azure-rest-api-specs#1429 * This commit for ADLS * Code gen for ADLS (part 2) * I missed this file! * Code gen for ADL (part 3) * Updating to the correct commit number: Azure/azure-rest-api-specs#1452 * Update Azure Batch to 5.1.0 (#3513) * Fixing publishing of symbols. Fixing issue# 3518 (#3519) * Fix Batch changelog references (#3517) * Do hydra to swagger specs migration for azure-sdk-for-net (#3482) * Do hydra to swagger specs migration for azure-sdk-for-net * Fix build failed problem * update the assemblyinfo file with the updated version 2.6.0 * Revert "update the assemblyinfo file with the updated version 2.6.0" This reverts commit a33beb0. * Updata the AssemblyFileVersion to 2.6.0.0 * revert package version * Make project files concise. * Add Provider test case * update the pr with new tests with session records * remove extra code * Fixing changelog.md * Fixing up changelog.md * [DataLake Analytics] Fixing changelog.md More descriptive and fixing indent issue that was not addressed in the previous commit * [DataLake Store] Fixing changelog.md More descriptive and fixing indent issue not addressed in a previous commit * [DataLake Analytics] Fixing changelog.md Adding an important note about retrieving a list of jobs and accounts * Revising custom Job Exists method * Ran into two regressions in JobOperationTests.cs * The custom Job Exists method is expecting a non-empty response body to validate non-existence. After speaking with a team member who implemented the service, I can no longer rely on this. Thus, I am removing the logic that checks for a non-empty response body. * Commenting out the test case for listing Jobs with a select parameter. This is currently being fixed. * Adding a TODO comment * Adding this as a reminder to re-enable the test
* In this already-merged PR (Azure#1429), Ben made some object-refactoring changes - one of them being to make all the fields in USqlJobProperties read-only as a good design decision. However, by making the fields statistics and debugData read-only, this caused some conflicts in the Powershell code. We need to pan out the design more carefully for these two fields with respect to their associated APIs ("/Jobs/{jobIdentity}/GetDebugDataPath" for the debugData field and "/Jobs/{jobIdentity}/GetStatistics" for the statistics field) before making them read-only. If I do not make the change proposed here, I would have to create two new cmdlets: one for getting the JobDataPath object and another for getting the JobStatistics object, which we do not want to do until we have a better understanding of their designs as a whole. Hence, the reverting change proposed here.
* Refactoring changes in order to match with Ben's refactoring changes * There is no functional change in this commit * This change is purely for refactoring related to Ben's refactoring changes (which has already been accepted): #1429 * Reordering fields in the example files to match with Ben's refactoring changes * Other minor changes or moving code around * [BUGFIX] - Reverting readOnly for two fields * In this already-merged PR (#1429), Ben made some object-refactoring changes - one of them being to make all the fields in USqlJobProperties read-only as a good design decision. However, by making the fields statistics and debugData read-only, this caused some conflicts in the Powershell code. We need to pan out the design more carefully for these two fields with respect to their associated APIs ("/Jobs/{jobIdentity}/GetDebugDataPath" for the debugData field and "/Jobs/{jobIdentity}/GetStatistics" for the statistics field) before making them read-only. If I do not make the change proposed here, I would have to create two new cmdlets: one for getting the JobDataPath object and another for getting the JobStatistics object, which we do not want to do until we have a better understanding of their designs as a whole. Hence, the reverting change proposed here.
* Code gen based on this PR: Azure/azure-rest-api-specs#1429 * Modifying the gulpfile to align with the directory revamp of the Swagger repo * Modifying the test files to work with the object-refactoring changes in the PR * Adding a few more tests to align with the well-covered tests for .NET SDK * Removing "import com.microsoft.azure.SubResource"
* Code gen for ADLA * This change is mapped to this PR: Azure/azure-rest-api-specs#1429 * This commit is for ADLA * Code gen for ADLS * This change is mapped to this PR: Azure/azure-rest-api-specs#1429 * This commit for ADLS * Code gen for ADLS (part 2) * I missed this file! * Code gen for ADL (part 3) * Updating to the correct commit number: Azure/azure-rest-api-specs#1452 * Update Azure Batch to 5.1.0 (#3513) * Fixing publishing of symbols. Fixing issue# 3518 (#3519) * Do hydra to swagger specs migration for azure-sdk-for-net (#3482) * Do hydra to swagger specs migration for azure-sdk-for-net * Fix build failed problem * update the assemblyinfo file with the updated version 2.6.0 * Revert "update the assemblyinfo file with the updated version 2.6.0" This reverts commit a33beb0. * Updata the AssemblyFileVersion to 2.6.0.0 * revert package version * Make project files concise. * Add Provider test case * update the pr with new tests with session records * remove extra code * Fix Batch changelog references (#3517) * Fixing changelog.md * Fixing up changelog.md * [DataLake Analytics] Fixing changelog.md More descriptive and fixing indent issue that was not addressed in the previous commit * [DataLake Store] Fixing changelog.md More descriptive and fixing indent issue not addressed in a previous commit * [DataLake Analytics] Fixing changelog.md Adding an important note about retrieving a list of jobs and accounts * Revising custom Job Exists method * Ran into two regressions in JobOperationTests.cs * The custom Job Exists method is expecting a non-empty response body to validate non-existence. After speaking with a team member who implemented the service, I can no longer rely on this. Thus, I am removing the logic that checks for a non-empty response body. * Commenting out the test case for listing Jobs with a select parameter. This is currently being fixed. * Adding a TODO comment * Adding this as a reminder to re-enable the test * [CODE GEN] - Reverting readOnly for two fields * Code gen based on this very descriptive PR: Azure/azure-rest-api-specs@e33ebf8 * [CODE GEN] - Updating the version (3.1.0 to 3.1.1)
…bject recursively (#1627) * Refactoring changes in order to match with Ben's refactoring changes * There is no functional change in this commit * This change is purely for refactoring related to Ben's refactoring changes (which has already been accepted): #1429 * Reordering fields in the example files to match with Ben's refactoring changes * Other minor changes or moving code around * Adding an innerError field to JobInnerError object There can be more levels of the innerError field in an JobInnerError object. There is currently only one level of nesting or innerError. Practically, however, the service returns up to two levels. Thus we made the change below: Currently we have: * JobErrorDetails * JobInnerError Now we have: * JobErrorDetails * JobInnerError * JobInnerSecond * JobInnerThird * Getting rid of JobInnerErrorSecond and JobInnerErrorThird * Getting rid of JobInnerErrorSecond and JobInnerErrorThird in place of a recursive definition * [BUGFIX] - Reverting readOnly for two fields * In this already-merged PR (#1429), Ben made some object-refactoring changes - one of them being to make all the fields in USqlJobProperties read-only as a good design decision. However, by making the fields statistics and debugData read-only, this caused some conflicts in the Powershell code. We need to pan out the design more carefully for these two fields with respect to their associated APIs ("/Jobs/{jobIdentity}/GetDebugDataPath" for the debugData field and "/Jobs/{jobIdentity}/GetStatistics" for the statistics field) before making them read-only. If I do not make the change proposed here, I would have to create two new cmdlets: one for getting the JobDataPath object and another for getting the JobStatistics object, which we do not want to do until we have a better understanding of their designs as a whole. Hence, the reverting change proposed here.
* Code gen for ADLA * This change is mapped to this PR: Azure/azure-rest-api-specs#1429 * This commit is for ADLA * Code gen for ADLS * This change is mapped to this PR: Azure/azure-rest-api-specs#1429 * This commit for ADLS * Code gen for ADLS (part 2) * I missed this file! * Code gen for ADL (part 3) * Updating to the correct commit number: Azure/azure-rest-api-specs#1452 * Update Azure Batch to 5.1.0 (#3513) * Fixing publishing of symbols. Fixing issue# 3518 (#3519) * Do hydra to swagger specs migration for azure-sdk-for-net (#3482) * Do hydra to swagger specs migration for azure-sdk-for-net * Fix build failed problem * update the assemblyinfo file with the updated version 2.6.0 * Revert "update the assemblyinfo file with the updated version 2.6.0" This reverts commit a33beb0. * Updata the AssemblyFileVersion to 2.6.0.0 * revert package version * Make project files concise. * Add Provider test case * update the pr with new tests with session records * remove extra code * Fix Batch changelog references (#3517) * Fixing changelog.md * Fixing up changelog.md * [DataLake Analytics] Fixing changelog.md More descriptive and fixing indent issue that was not addressed in the previous commit * [DataLake Store] Fixing changelog.md More descriptive and fixing indent issue not addressed in a previous commit * [DataLake Analytics] Fixing changelog.md Adding an important note about retrieving a list of jobs and accounts * Revising custom Job Exists method * Ran into two regressions in JobOperationTests.cs * The custom Job Exists method is expecting a non-empty response body to validate non-existence. After speaking with a team member who implemented the service, I can no longer rely on this. Thus, I am removing the logic that checks for a non-empty response body. * Commenting out the test case for listing Jobs with a select parameter. This is currently being fixed. * Adding a TODO comment * Adding this as a reminder to re-enable the test * [CODE GEN] - Reverting readOnly for two fields * Code gen based on this very descriptive PR: Azure/azure-rest-api-specs@e33ebf8 * [CODE GEN] - Updating the version (3.1.0 to 3.1.1) * [CODE REFACTOR] - Removing flaky test * Removing test case: DataLakeUploader_FreshFolderUploadDownload() * Note: In the near future, the custom ADLS Filesystem code will be replaced with a new SDK that has its own set of test cases. The account management side of ADLS will still remain, however. * [DOC] - Updating the changelog to describe 3.1.1-preview * [CODE REFACTOR] - Skipping flaky test * Skipping the flaky test as opposed to commenting it out * Also, I forgot to replace a SessionRecord file from last update, so I doing it now
* Code gen for ADLA * This change is mapped to this PR: Azure/azure-rest-api-specs#1429 * This commit is for ADLA * Code gen for ADLS * This change is mapped to this PR: Azure/azure-rest-api-specs#1429 * This commit for ADLS * Code gen for ADLS (part 2) * I missed this file! * Code gen for ADL (part 3) * Updating to the correct commit number: Azure/azure-rest-api-specs#1452 * Update Azure Batch to 5.1.0 (#3513) * Fixing publishing of symbols. Fixing issue# 3518 (#3519) * Do hydra to swagger specs migration for azure-sdk-for-net (#3482) * Do hydra to swagger specs migration for azure-sdk-for-net * Fix build failed problem * update the assemblyinfo file with the updated version 2.6.0 * Revert "update the assemblyinfo file with the updated version 2.6.0" This reverts commit a33beb0. * Updata the AssemblyFileVersion to 2.6.0.0 * revert package version * Make project files concise. * Add Provider test case * update the pr with new tests with session records * remove extra code * Fix Batch changelog references (#3517) * Fixing changelog.md * Fixing up changelog.md * [DataLake Analytics] Fixing changelog.md More descriptive and fixing indent issue that was not addressed in the previous commit * [DataLake Store] Fixing changelog.md More descriptive and fixing indent issue not addressed in a previous commit * [DataLake Analytics] Fixing changelog.md Adding an important note about retrieving a list of jobs and accounts * Revising custom Job Exists method * Ran into two regressions in JobOperationTests.cs * The custom Job Exists method is expecting a non-empty response body to validate non-existence. After speaking with a team member who implemented the service, I can no longer rely on this. Thus, I am removing the logic that checks for a non-empty response body. * Commenting out the test case for listing Jobs with a select parameter. This is currently being fixed. * Adding a TODO comment * Adding this as a reminder to re-enable the test * [CODE GEN] - Reverting readOnly for two fields * Code gen based on this very descriptive PR: Azure/azure-rest-api-specs@e33ebf8 * [CODE GEN] - Updating the version (3.1.0 to 3.1.1) * [CODE REFACTOR] - Removing flaky test * Removing test case: DataLakeUploader_FreshFolderUploadDownload() * Note: In the near future, the custom ADLS Filesystem code will be replaced with a new SDK that has its own set of test cases. The account management side of ADLS will still remain, however. * [DOC] - Updating the changelog to describe 3.1.1-preview * [CODE REFACTOR] - Skipping flaky test * Skipping the flaky test as opposed to commenting it out * Also, I forgot to replace a SessionRecord file from last update, so I doing it now * [BUG FIX] - Making JobInnerError object recursively defined * Based on this PR: Azure/azure-rest-api-specs#1627 * We are essentially adding one more property to the JobInnerError object (recursively) in order to capture the additional error information that's returned by the service * This added property has been tested successfully by a team member
…and object refactoring (#1893) * Adding a parameter to catalogs, Pipeline and Recurrence information, and object refactoring * Catalogs.java: added a parameter, called basic, to indicate that you want basic information rather than nested information about a table (i.e., list tables method) for performance increase * Adding Pipeline and Recurrence information to jobs * Added methods to get such information * Refactoring and comment changes * Code gen for ADLS * Refactoring and comment changes * [CODE GEN] - BREAKING CHANGE: Adl object refactor * Code gen based on this PR: Azure/azure-rest-api-specs#1429 * Modifying the gulpfile to align with the directory revamp of the Swagger repo * Modifying the test files to work with the object-refactoring changes in the PR * Adding a few more tests to align with the well-covered tests for .NET SDK * Removing "import com.microsoft.azure.SubResource" * [BUGFIX] - Fixing up and commenting out tests * For JobOperationsTests, I found a server side bug while testing this call in the .NET SDK, so I was not surprised to see that it fails here as well. * For CatalogOperationsTests, I commented out "BUCKETS" from the script because apparently that keyword is deprecated. Furthermore, I commented out the "canCreateUpdateDeleteSecretsAndCredentials" test because the scripts that are being used are deprecated. According to the error message, I must use Azure Powershell cmdlets for such opeartions. I am not surprised at this because, to my knowledge, it's been a while since this test case was run. I will look into how I can test this in the near future. * Other than that, I made sure that JobOperationsTests as well as AccountOperationsTests (for both ADLA and ADLS) passed. * [CODE REFACTOR] - Removing an import to pass style check * Removing the unused import java.io.IOException across files to pass Maven style check * [CODE REFACTOR] - Generating UUID from SdkContext * This is so that Travis passes! * [FILE REFACTOR] - Updating Session Records * [BUGFIX] - Making sure variables are caught in recording * Three variables in DataLakeAnalyticsMangementTestBase.java weren't recorded * [COMMENT] - The moving of the declaration of variables * [COMMENT REFACTORING] - Minor refactoring * [CODE REFACTOR] - Adding a small sleep for better test results * [CODE REFACTOR] - Making the sleep longer * Apparently the cache is restored every 5 minutes
…and object refactoring (#1893) * Adding a parameter to catalogs, Pipeline and Recurrence information, and object refactoring * Catalogs.java: added a parameter, called basic, to indicate that you want basic information rather than nested information about a table (i.e., list tables method) for performance increase * Adding Pipeline and Recurrence information to jobs * Added methods to get such information * Refactoring and comment changes * Code gen for ADLS * Refactoring and comment changes * [CODE GEN] - BREAKING CHANGE: Adl object refactor * Code gen based on this PR: Azure/azure-rest-api-specs#1429 * Modifying the gulpfile to align with the directory revamp of the Swagger repo * Modifying the test files to work with the object-refactoring changes in the PR * Adding a few more tests to align with the well-covered tests for .NET SDK * Removing "import com.microsoft.azure.SubResource" * [BUGFIX] - Fixing up and commenting out tests * For JobOperationsTests, I found a server side bug while testing this call in the .NET SDK, so I was not surprised to see that it fails here as well. * For CatalogOperationsTests, I commented out "BUCKETS" from the script because apparently that keyword is deprecated. Furthermore, I commented out the "canCreateUpdateDeleteSecretsAndCredentials" test because the scripts that are being used are deprecated. According to the error message, I must use Azure Powershell cmdlets for such opeartions. I am not surprised at this because, to my knowledge, it's been a while since this test case was run. I will look into how I can test this in the near future. * Other than that, I made sure that JobOperationsTests as well as AccountOperationsTests (for both ADLA and ADLS) passed. * [CODE REFACTOR] - Removing an import to pass style check * Removing the unused import java.io.IOException across files to pass Maven style check * [CODE REFACTOR] - Generating UUID from SdkContext * This is so that Travis passes! * [FILE REFACTOR] - Updating Session Records * [BUGFIX] - Making sure variables are caught in recording * Three variables in DataLakeAnalyticsMangementTestBase.java weren't recorded * [COMMENT] - The moving of the declaration of variables * [COMMENT REFACTORING] - Minor refactoring * [CODE REFACTOR] - Adding a small sleep for better test results * [CODE REFACTOR] - Making the sleep longer * Apparently the cache is restored every 5 minutes
…and object refactoring (#1893) * Adding a parameter to catalogs, Pipeline and Recurrence information, and object refactoring * Catalogs.java: added a parameter, called basic, to indicate that you want basic information rather than nested information about a table (i.e., list tables method) for performance increase * Adding Pipeline and Recurrence information to jobs * Added methods to get such information * Refactoring and comment changes * Code gen for ADLS * Refactoring and comment changes * [CODE GEN] - BREAKING CHANGE: Adl object refactor * Code gen based on this PR: Azure/azure-rest-api-specs#1429 * Modifying the gulpfile to align with the directory revamp of the Swagger repo * Modifying the test files to work with the object-refactoring changes in the PR * Adding a few more tests to align with the well-covered tests for .NET SDK * Removing "import com.microsoft.azure.SubResource" * [BUGFIX] - Fixing up and commenting out tests * For JobOperationsTests, I found a server side bug while testing this call in the .NET SDK, so I was not surprised to see that it fails here as well. * For CatalogOperationsTests, I commented out "BUCKETS" from the script because apparently that keyword is deprecated. Furthermore, I commented out the "canCreateUpdateDeleteSecretsAndCredentials" test because the scripts that are being used are deprecated. According to the error message, I must use Azure Powershell cmdlets for such opeartions. I am not surprised at this because, to my knowledge, it's been a while since this test case was run. I will look into how I can test this in the near future. * Other than that, I made sure that JobOperationsTests as well as AccountOperationsTests (for both ADLA and ADLS) passed. * [CODE REFACTOR] - Removing an import to pass style check * Removing the unused import java.io.IOException across files to pass Maven style check * [CODE REFACTOR] - Generating UUID from SdkContext * This is so that Travis passes! * [FILE REFACTOR] - Updating Session Records * [BUGFIX] - Making sure variables are caught in recording * Three variables in DataLakeAnalyticsMangementTestBase.java weren't recorded * [COMMENT] - The moving of the declaration of variables * [COMMENT REFACTORING] - Minor refactoring * [CODE REFACTOR] - Adding a small sleep for better test results * [CODE REFACTOR] - Making the sleep longer * Apparently the cache is restored every 5 minutes
This checklist is used to make sure that common issues in a pull request are addressed. This will expedite the process of getting your pull request merged and avoid extra work on your part to fix issues discovered during the review process.
Description
This PR addresses the following issues by restructuring our object models, which is a breaking change
Specifically we have done the following:
JobInformation
intoCreateJobParameters
,JobInformationBasic
andJobInformation
DataLakeAnalyticsAccount
andDataLakeAnalyticsAccountProperties
into those and BasicVersions
DataLakeStoreAccount
andDataLakeStoreAccountProperties
into those and Basic VersionsPR information
api-version
in the path should match theapi-version
in the spec).Quality of Swagger