Ingest updates for split #198
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…KustoConnectionStringBuilder, and not a client struct.
* In addition, passing a default database and table for ingestion is not necessary anymore, and can be done using Options.
```go
// before:
queryClient := kusto.New("https://ingest-somecluster.kusto.windows.net")
client := ingest.New(quetryClient, "some-db", "some-table")
// after:
client := azkustoingest.New("https://ingest-somecluster.kusto.windows.net", azkustoingest.WithDefaultDatabase("someDb"), azkustoingest.WithDefaultTable("someTable"))
```
* Added autocorrection for endpoints for ingest clients. When creating a client, the "ingest-" will be added or removed as needed. To avoid this behavior, use the `azkustoingest.WithoutEndpointCorrection()` option.
* ManagedStreamingClient constructor now only requires the query endpoint, and will infer the ingest endpoint from it. If you want to use a different endpoint, use the `azkustoingest.WithCustomIngestConnectionString()` option.
* Removed the old deprecated Stream() method on queued ingest client, instead use azkustoingest.NewStreaming() or azkustoingest.NewManaged() for proper streaming ingest client.
* Removed `QueryIngestion()` option for Query client. If you want to perform commands against the dm, create a query client with the "ingest-" endpoint.
Codecov Report
@@ Coverage Diff @@
## the-big-split #198 +/- ##
=================================================
+ Coverage 49.56% 50.78% +1.21%
=================================================
Files 54 54
Lines 5909 5919 +10
=================================================
+ Hits 2929 3006 +77
+ Misses 2765 2699 -66
+ Partials 215 214 -1
|
yogilad
reviewed
Aug 3, 2023
azkustoingest/common.go
Outdated
| @@ -0,0 +1,109 @@ | |||
| package azkustoingest | |||
There was a problem hiding this comment.
Why did you choose the name "Common"?
Isn't Options more suitable
There was a problem hiding this comment.
renamed to ingestion_options
azkustoingest/streaming.go
Outdated
| @@ -171,6 +189,10 @@ func (i *Streaming) newProp() properties.All { | |||
| } | |||
| } | |||
|
|
|||
| func (i *Streaming) QueryClient() QueryClient { | |||
There was a problem hiding this comment.
Why is this a public API?
Do we need it for something other than tests?
There was a problem hiding this comment.
Getting access to the client is useful, otherwise to run commands like changing the batching will require creating a new one.
It does save work on tests, but I think it's useful regardless.
azkustoingest/ingest.go
Outdated
| ) | ||
|
|
||
| type Ingestor interface { | ||
| io.Closer | ||
| FromFile(ctx context.Context, fPath string, options ...FileOption) (*Result, error) | ||
| FromReader(ctx context.Context, reader io.Reader, options ...FileOption) (*Result, error) | ||
| QueryClient() QueryClient |
There was a problem hiding this comment.
Why does the ingest API publicly expose a Query Client?
Co-authored-by: Yochai Gilad <[email protected]>
Co-authored-by: Yochai Gilad <[email protected]>
AsafMah
added a commit
that referenced
this pull request
Mar 11, 2024
* Split & rename modules * fix go.mod * rename to azkustodata and azkustoingest * update * Update build.yml * build.yml changes * fixes * build * fixed merge * try * restore dataset.csv * ### Added - azidentity package dependency ### Changed - Update azkustoingest/test/etoe/etoe_test.go to use azkustoingest.FileOption ### Fixed ### Removed ### Security * Fixed * Centralize deps * Fixed once_test.go * Fixed once_test.go * Fixed once_test.go * Minor fixes * Solved flakeyness maybe * remove replace * no replacements * Will this fail? * Will this fail? * Will this fail? * Will this fail? * Will this fail? * Will this fail? * Ok check 1 * Tidy * TidyDoc fixes * Split e2e tests * Updated deps * go work sync * this is not good * Test change * Test change * PR fixes * Fix more * Stop more queries * Stop more queries * Check if problem test * Rename package * Rename package * Rename package * Gofmt * Fixed error * tidy * Tidy part 2 * go.sum * Ingest updates for split (#198) * * [BREAKING] [MAJOR] Constructing ingest clients is now done using a KustoConnectionStringBuilder, and not a client struct. * In addition, passing a default database and table for ingestion is not necessary anymore, and can be done using Options. ```go // before: queryClient := kusto.New("https://ingest-somecluster.kusto.windows.net") client := ingest.New(quetryClient, "some-db", "some-table") // after: client := azkustoingest.New("https://ingest-somecluster.kusto.windows.net", azkustoingest.WithDefaultDatabase("someDb"), azkustoingest.WithDefaultTable("someTable")) ``` * Added autocorrection for endpoints for ingest clients. When creating a client, the "ingest-" will be added or removed as needed. To avoid this behavior, use the `azkustoingest.WithoutEndpointCorrection()` option. * ManagedStreamingClient constructor now only requires the query endpoint, and will infer the ingest endpoint from it. If you want to use a different endpoint, use the `azkustoingest.WithCustomIngestConnectionString()` option. * Removed the old deprecated Stream() method on queued ingest client, instead use azkustoingest.NewStreaming() or azkustoingest.NewManaged() for proper streaming ingest client. * Removed `QueryIngestion()` option for Query client. If you want to perform commands against the dm, create a query client with the "ingest-" endpoint. * Tests and docs * More tests * fmt * fmt * Try to improve build errors * f * f * Try again * Try again * Try again * Fix * Update README.md Co-authored-by: Yochai Gilad <[email protected]> * Update azkustoingest/common.go Co-authored-by: Yochai Gilad <[email protected]> * Rename * Remove queryclient * Remove queryclient * Remove queryclient * Remove queryclient --------- Co-authored-by: Yochai Gilad <[email protected]> * Merge branch 'master' into query-v2 * merge from main * Updated deps * Fixed * Merge fixes * Merge fixes * Query v2 (#216) * Frame definitions * Working json (kinda) * Improved values * Improved values * Improved values * Compilation errors * More fixes * meanwhile commit * merge from main * Reworked kusto types - No more duplications between packages. Removed old system of query building. Fixed tests, etc. * Query v2 - tables, frames, all * Added query * More tests * Working tests * More tests * Something is off * Something is off * Fixed error * More tests, more tables * Fixed tests, aligned types, etc * -int types can be deserialized to struct (with error on overflow) -fixed secondary tables * Easy row consumption and op for errors * Some docs * Examples and tests * Fixed tests * Fixed data races. Removed fixed table (for now) * Fixed deadlock + parallel tests * Fixed deadlocks * Fixed deadlocks * Rework tables into results * Big refactor - split common data from v2 specific * More v1 stuff * More v1 stuff * Moving stuff around for compile errors * Fixed tests * Fixed recursive lock issue * Fixed recursive lock issue * Fixed recursive lock issue * Fixed missing break * Fixed missing break * Fixed test * Make sure the correct error is always called, without race condition * Make sure the correct error is always called, without race condition * Make sure the correct error is always called, without race condition * upgrade * Merge fixes * Merge fixes * Tests and fixes * Tests and fixes * Split v2 tables to streaming and non-streaming * Fixed tests * Renames * Fixed error * Renaming * More api and partial results test * Fixed read * Some docs * -Renames and add apis -Fixed partial Results text -Fixed full errors tests * full dataset tests * Fixed tests * Handle partial errors in v2 & fixed number handling in v1 * Naming and comments * PR fixes * PR fixes * Added error description and table progress * Added error description and table progress * Fix tests * PR fixes * Fix tests * Removed more old code, fixed tests, still wip * Fixed all tests * Fixed + added benchmark * Fixed ingest tests by relocating mock * Fixed tests * More fixed tests * Fixed race * typo * remove todo (for now) * Cherrypick changes from other branch * -Simplify and rename * PR Changes: - Simplify code to handle only what needed - Rename ordinal to index - Some comments * Fixed samples * Fixed cycle * Removed old full dataset things * Renames and bug fixes * Aligned types, made full tables make more sense. * Properly close tables, fixed deadlocks and tests. * Fixed ingest tests * Fixed test * build.yml * Updated sample app * Improve allocations and fix benchmarks * Improve values system * -Reduced code dups in structs -Fixed edge cases * Jsontest - correct order of params * cache decodeToStruct result * PR stuff * PR stuff * PR changes - part 1 * PR Changes - renames * Readne * Lock Cache * Docs and migration for 1.0.0 (#225) * Bump golang.org/x/crypto from 0.14.0 to 0.17.0 (#219) Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.14.0 to 0.17.0. - [Commits](golang/crypto@v0.14.0...v0.17.0) --- updated-dependencies: - dependency-name: golang.org/x/crypto dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump golang.org/x/crypto from 0.14.0 to 0.17.0 in /quickstart (#218) Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.14.0 to 0.17.0. - [Commits](golang/crypto@v0.14.0...v0.17.0) --- updated-dependencies: - dependency-name: golang.org/x/crypto dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Changed just sstream to false. the rest was fine (#220) * Update issue templates (#223) * Update build.yml * Update build.yml (#224) * T ronmoneta/compression auto detect fix (#222) * Changed just sstream to false. the rest was fine * Switched rows * New doc files and Migration guide * Update MIGRATION.md Co-authored-by: ohad bitton <[email protected]> * Update MIGRATION.md Co-authored-by: ohad bitton <[email protected]> * Update MIGRATION.md Co-authored-by: Yochai Gilad <[email protected]> * New doc files and Migration guide * Last fixes * azkustodata tidy * version fixing * version fixing --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: t-ronmoneta <[email protected]> Co-authored-by: ohad bitton <[email protected]> Co-authored-by: Yochai Gilad <[email protected]> * v1.0.0-preview * last adjustments * last adjustments * last adjustments * last adjustments * last adjustments --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Yochai Gilad <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: t-ronmoneta <[email protected]> Co-authored-by: ohad bitton <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
[BREAKING] [MAJOR] Constructing ingest clients is now done using a KustoConnectionStringBuilder, and not a client struct.
In addition, passing a default database and table for ingestion is not necessary anymore, and can be done using Options.
Added autocorrection for endpoints for ingest clients. When creating a client, the "ingest-" will be added or removed as needed. To avoid this behavior, use the
azkustoingest.WithoutEndpointCorrection()option.ManagedStreamingClient constructor now only requires the query endpoint, and will infer the ingest endpoint from it. If you want to use a different endpoint, use the
azkustoingest.WithCustomIngestConnectionString()option.Removed the old deprecated Stream() method on queued ingest client, instead use azkustoingest.NewStreaming() or azkustoingest.NewManaged() for proper streaming ingest client.
Removed
QueryIngestion()option for Query client. If you want to perform commands against the dm, create a query client with the "ingest-" endpoint.