Skip to content

Commit

Permalink
Potentially Breaking Change: Update ADLA catalog (Azure#3118)
Browse files Browse the repository at this point in the history
* Update to latest swagger spec for ADL

This change affects catalog functionality:
* Add new package catalog item type
* Update to allow for listing of common catalog items from a base
database:
- Table
- TableValuedFunction
- TableStatistics
- Views

Updates for tests and versions.

comment out new tests for ADL

doing this until i can run tests in my local environment.

Update tests and recordings.

Actually copying the right file to the right place.

update changelog.

* Cleanup ADL after migration to dev17

This cleans up files that are no longer used and updates csproj files to
remove commented out sections

* Added a number of null checks to ADLS client

We were getting some null reference exceptions on occasion so ensuring
none of these nested properties are null before indexing to them.

* Move package ID to the top to address PR.
  • Loading branch information
begoldsm authored and shahabhijeet committed Apr 26, 2017
1 parent e5f49e5 commit 178ed88
Show file tree
Hide file tree
Showing 23 changed files with 7,491 additions and 4,024 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,16 @@ public void GetCatalogItemsTest()
// look for the table we created
Assert.True(tableListResponse.Any(table => table.Name.Equals(commonData.TableName)));

// get the table list in just the db
tableListResponse = clientToUse.Catalog.ListTablesByDatabase(
commonData.SecondDataLakeAnalyticsAccountName,
commonData.DatabaseName);

Assert.True(tableListResponse.Count() >= 1);

// look for the table we created
Assert.True(tableListResponse.Any(table => table.Name.Equals(commonData.TableName)));

// Get the specific table as well
var tableGetResponse = clientToUse.Catalog.GetTable(
commonData.SecondDataLakeAnalyticsAccountName,
Expand All @@ -72,6 +82,16 @@ public void GetCatalogItemsTest()
// look for the tvf we created
Assert.True(tvfListResponse.Any(tvf => tvf.Name.Equals(commonData.TvfName)));

// get tvf list in the database
tvfListResponse = clientToUse.Catalog.ListTableValuedFunctionsByDatabase(
commonData.SecondDataLakeAnalyticsAccountName,
commonData.DatabaseName);

Assert.True(tvfListResponse.Count() >= 1);

// look for the tvf we created
Assert.True(tvfListResponse.Any(tvf => tvf.Name.Equals(commonData.TvfName)));

// Get the specific TVF as well
var tvfGetResponse = clientToUse.Catalog.GetTableValuedFunction(
commonData.SecondDataLakeAnalyticsAccountName,
Expand All @@ -89,6 +109,16 @@ public void GetCatalogItemsTest()
// look for the view we created
Assert.True(viewListResponse.Any(view => view.Name.Equals(commonData.ViewName)));

// get the view list from just the database
viewListResponse = clientToUse.Catalog.ListViewsByDatabase(
commonData.SecondDataLakeAnalyticsAccountName,
commonData.DatabaseName);

Assert.True(viewListResponse.Count() >= 1);

// look for the view we created
Assert.True(viewListResponse.Any(view => view.Name.Equals(commonData.ViewName)));

// Get the specific view as well
var viewGetResponse = clientToUse.Catalog.GetView(
commonData.SecondDataLakeAnalyticsAccountName,
Expand Down

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ internal static class DataLakeAnalyticsCustomizationHelper
/// This constant is used as the default package version to place in the user agent.
/// It should mirror the package version in the project.json file.
/// </summary>
internal const string PackageVersion = "2.1.1-preview";
internal const string PackageVersion = "2.2.0-preview";

internal const string DefaultAdlaDnsSuffix = "azuredatalakeanalytics.net";

Expand Down

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
Expand Up @@ -12,20 +12,14 @@ namespace Microsoft.Azure.Management.DataLake.Analytics.Models
using Microsoft.Azure.Management;
using Microsoft.Azure.Management.DataLake;
using Microsoft.Azure.Management.DataLake.Analytics;
using Newtonsoft.Json;
using Newtonsoft.Json.Converters;
using System.Runtime;
using System.Runtime.Serialization;

/// <summary>
/// Defines values for FileType.
/// </summary>
[JsonConverter(typeof(StringEnumConverter))]
public enum FileType
public static class FileType
{
[EnumMember(Value = "Assembly")]
Assembly,
[EnumMember(Value = "Resource")]
Resource
public const string Assembly = "Assembly";
public const string Resource = "Resource";
public const string Nodeploy = "Nodeploy";
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -32,12 +32,12 @@ public USqlAssemblyFileInfo()
/// Initializes a new instance of the USqlAssemblyFileInfo class.
/// </summary>
/// <param name="type">the assembly file type. Possible values include:
/// 'Assembly', 'Resource'</param>
/// 'Assembly', 'Resource', 'Nodeploy'</param>
/// <param name="originalPath">the the original path to the assembly
/// file.</param>
/// <param name="contentPath">the the content path to the assembly
/// file.</param>
public USqlAssemblyFileInfo(FileType? type = default(FileType?), string originalPath = default(string), string contentPath = default(string))
public USqlAssemblyFileInfo(string type = default(string), string originalPath = default(string), string contentPath = default(string))
{
Type = type;
OriginalPath = originalPath;
Expand All @@ -52,10 +52,10 @@ public USqlAssemblyFileInfo()

/// <summary>
/// Gets or sets the assembly file type. Possible values include:
/// 'Assembly', 'Resource'
/// 'Assembly', 'Resource', 'Nodeploy'
/// </summary>
[JsonProperty(PropertyName = "type")]
public FileType? Type { get; set; }
public string Type { get; set; }

/// <summary>
/// Gets or sets the the original path to the assembly file.
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License. See License.txt in the project root for
// license information.
//
// Code generated by Microsoft (R) AutoRest Code Generator 1.0.1.0
// Changes may cause incorrect behavior and will be lost if the code is
// regenerated.

namespace Microsoft.Azure.Management.DataLake.Analytics.Models
{
using Microsoft.Azure;
using Microsoft.Azure.Management;
using Microsoft.Azure.Management.DataLake;
using Microsoft.Azure.Management.DataLake.Analytics;
using Newtonsoft.Json;
using System.Linq;

/// <summary>
/// A Data Lake Analytics catalog U-SQL package item.
/// </summary>
public partial class USqlPackage : CatalogItem
{
/// <summary>
/// Initializes a new instance of the USqlPackage class.
/// </summary>
public USqlPackage()
{
CustomInit();
}

/// <summary>
/// Initializes a new instance of the USqlPackage class.
/// </summary>
/// <param name="computeAccountName">the name of the Data Lake
/// Analytics account.</param>
/// <param name="version">the version of the catalog item.</param>
/// <param name="databaseName">the name of the database containing the
/// package.</param>
/// <param name="schemaName">the name of the schema associated with
/// this package and database.</param>
/// <param name="name">the name of the package.</param>
/// <param name="definition">the definition of the package.</param>
public USqlPackage(string computeAccountName = default(string), System.Guid? version = default(System.Guid?), string databaseName = default(string), string schemaName = default(string), string name = default(string), string definition = default(string))
: base(computeAccountName, version)
{
DatabaseName = databaseName;
SchemaName = schemaName;
Name = name;
Definition = definition;
CustomInit();
}

/// <summary>
/// An initialization method that performs custom operations like setting defaults
/// </summary>
partial void CustomInit();

/// <summary>
/// Gets or sets the name of the database containing the package.
/// </summary>
[JsonProperty(PropertyName = "databaseName")]
public string DatabaseName { get; set; }

/// <summary>
/// Gets or sets the name of the schema associated with this package
/// and database.
/// </summary>
[JsonProperty(PropertyName = "schemaName")]
public string SchemaName { get; set; }

/// <summary>
/// Gets or sets the name of the package.
/// </summary>
[JsonProperty(PropertyName = "packageName")]
public string Name { get; set; }

/// <summary>
/// Gets or sets the definition of the package.
/// </summary>
[JsonProperty(PropertyName = "definition")]
public string Definition { get; set; }

}
}
Original file line number Diff line number Diff line change
@@ -1,66 +1,14 @@
<Project Sdk="Microsoft.NET.Sdk">
<Import Project="$([MSBuild]::GetPathOfFileAbove('AzSdk.reference.props'))" />
<PropertyGroup>
<PackageId>Microsoft.Azure.Management.DataLake.Analytics</PackageId>
<Description>Provides Data Lake Analytics account, job and catalog management capabilities for Microsoft Azure.</Description>
<VersionPrefix>2.1.0-preview</VersionPrefix>
<VersionPrefix>2.2.0-preview</VersionPrefix>
<AssemblyName>Microsoft.Azure.Management.DataLake.Analytics</AssemblyName>

<!--<Authors>Microsoft</Authors>
<TargetFrameworks>net45;netstandard1.5;netstandard1.1</TargetFrameworks>
<DelaySign>true</DelaySign>
<AssemblyOriginatorKeyFile>../../../../tools/MSSharedLibKey.snk</AssemblyOriginatorKeyFile>
<SignAssembly>true</SignAssembly>-->
<PackageId>Microsoft.Azure.Management.DataLake.Analytics</PackageId>
<PackageTags>Microsoft Azure Data Lake Analytics management;DataLakeAnalytics;Data Lake Analytics management;REST HTTP client;windowsazureofficial;netcore451511</PackageTags>
<PackageReleaseNotes>See https://github.com/azure/azure-sdk-for-net/blob/AutoRest/src/ResourceManagement/DataLake.Analytics/changelog.md for release notes.</PackageReleaseNotes>
<PackageIconUrl>http://go.microsoft.com/fwlink/?LinkID=288890</PackageIconUrl>
<PackageProjectUrl>https://github.com/Azure/azure-sdk-for-net</PackageProjectUrl>
<PackageLicenseUrl>https://raw.githubusercontent.com/Microsoft/dotnet/master/LICENSE</PackageLicenseUrl>
<PackageRequireLicenseAcceptance>true</PackageRequireLicenseAcceptance>
<!--<PackageTargetFallback Condition=" '$(TargetFramework)' == 'netstandard1.5' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
<NetStandardImplicitPackageVersion Condition=" '$(TargetFramework)' == 'netstandard1.5' ">1.6.1</NetStandardImplicitPackageVersion>
<PackageTargetFallback Condition=" '$(TargetFramework)' == 'netstandard1.1' ">$(PackageTargetFallback);dnxcore50</PackageTargetFallback>
<NetStandardImplicitPackageVersion Condition=" '$(TargetFramework)' == 'netstandard1.1' ">1.6.1</NetStandardImplicitPackageVersion>-->
<GenerateAssemblyTitleAttribute>false</GenerateAssemblyTitleAttribute>
<GenerateAssemblyDescriptionAttribute>false</GenerateAssemblyDescriptionAttribute>
<GenerateAssemblyVersionAttribute>false</GenerateAssemblyVersionAttribute>
<GenerateAssemblyFileVersionAttribute>false</GenerateAssemblyFileVersionAttribute>
<GenerateAssemblyConfigurationAttribute>false</GenerateAssemblyConfigurationAttribute>
<GenerateAssemblyCompanyAttribute>false</GenerateAssemblyCompanyAttribute>
<GenerateAssemblyProductAttribute>false</GenerateAssemblyProductAttribute>
<GenerateAssemblyCopyrightAttribute>false</GenerateAssemblyCopyrightAttribute>
<GenerateNeutralResourcesLanguageAttribute>false</GenerateNeutralResourcesLanguageAttribute>
<PackageReleaseNotes>See https://github.com/azure/azure-sdk-for-net/blob/vs17Dev/src/SDKs/DataLake.Analytics/changelog.md for release notes.</PackageReleaseNotes>
</PropertyGroup>

<PropertyGroup>
<TargetFrameworks>net452;netstandard1.4</TargetFrameworks>
</PropertyGroup>



<!--
<ItemGroup>
<PackageReference Include="Microsoft.Rest.ClientRuntime.Azure" Version="[3.3.5,4.0.0)" />
<PackageReference Include="Microsoft.Rest.ClientRuntime" Version="[2.3.5,3.0.0)" />
</ItemGroup>
<ItemGroup Condition=" '$(TargetFramework)' == 'net45' ">
<Reference Include="System" />
<Reference Include="Microsoft.CSharp" />
</ItemGroup>
<ItemGroup Condition=" '$(TargetFramework)' == 'netstandard1.5' ">
<PackageReference Include="Microsoft.NETCore.Platforms" Version="1.1.0" />
<PackageReference Include="System.Diagnostics.Tools" Version="4.3.0" />
<PackageReference Include="System.Net.Http" Version="4.3.0" />
<PackageReference Include="System.Runtime.Serialization.Primitives" Version="4.1.1" />
<PackageReference Include="System.Threading.Tasks" Version="4.3.0" />
</ItemGroup>
<ItemGroup Condition=" '$(TargetFramework)' == 'netstandard1.1' ">
<PackageReference Include="Microsoft.NETCore.Platforms" Version="1.1.0" />
<PackageReference Include="System.Runtime.Serialization.Primitives" Version="4.1.1" />
</ItemGroup>
-->
</Project>
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
[assembly: AssemblyDescription("Provides Microsoft Azure Data Lake Analytics management operations including account, catalog and job management.")]

[assembly: AssemblyVersion("2.0.0.0")]
[assembly: AssemblyFileVersion("2.1.1.0")]
[assembly: AssemblyFileVersion("2.2.0.0")]
[assembly: AssemblyConfiguration("")]
[assembly: AssemblyCompany("Microsoft")]
[assembly: AssemblyProduct("Azure .NET SDK")]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@
@echo off
setlocal

set accountSpecFile="https://raw.githubusercontent.com/Azure/azure-rest-api-specs/0ade6547eff89bab427eb400ebc5da870f2c5bd0/arm-datalake-analytics/account/2016-11-01/swagger/account.json"
set jobSpecFile="https://raw.githubusercontent.com/Azure/azure-rest-api-specs/0ade6547eff89bab427eb400ebc5da870f2c5bd0/arm-datalake-analytics/job/2016-11-01/swagger/job.json"
set catalogSpecFile="https://raw.githubusercontent.com/Azure/azure-rest-api-specs/0ade6547eff89bab427eb400ebc5da870f2c5bd0/arm-datalake-analytics/catalog/2016-11-01/swagger/catalog.json"
set accountSpecFile="https://raw.githubusercontent.com/Azure/azure-rest-api-specs/12a6c2299296b6877fa1e5a03ed42752445877dc/arm-datalake-analytics/account/2016-11-01/swagger/account.json"
set jobSpecFile="https://raw.githubusercontent.com/Azure/azure-rest-api-specs/12a6c2299296b6877fa1e5a03ed42752445877dc/arm-datalake-analytics/job/2016-11-01/swagger/job.json"
set catalogSpecFile="https://raw.githubusercontent.com/Azure/azure-rest-api-specs/12a6c2299296b6877fa1e5a03ed42752445877dc/arm-datalake-analytics/catalog/2016-11-01/swagger/catalog.json"

set repoRoot=%~dp0..\..\..\..
set generateFolder=%~dp0Generated
Expand Down
9 changes: 9 additions & 0 deletions src/SDKs/DataLake.Analytics/changelog.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,14 @@
## Microsoft.Azure.Management.DataLake.Analytics release notes

### Changes in 2.2.0-preview
- Introduces the new package item type for catalog items.
- Switched the FileType enum to a string since this field is read only and will have new types added and changed with some regularity.
- Allows for listing the following catalog items from their parent's parent item (the database)
- Table
- view
- Table valued function
- Table statistics

### Changes in 2.1.1-preview
- Update underlying AutoRest framework to the latest. This cleans up a lot of the generated code, making it easier to read.
- The property `FirewallAllowAzureIps` in the Account object is currently ignored by the service and will not have a value when calling get. This property will be enabled and honored by the service in a future update.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@
</PropertyGroup>

<ItemGroup>
<!--<PackageReference Include="Microsoft.Azure.Management.DataLake.Store" Version="2.1.1-preview" />-->
<ProjectReference Include="..\Management.DataLake.Store\Microsoft.Azure.Management.DataLake.Store.csproj" />
</ItemGroup>

Expand Down
Loading

0 comments on commit 178ed88

Please sign in to comment.