Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement Table.replace for the database backend #8986

Merged
merged 94 commits into from
Feb 29, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
94 commits
Select commit Hold shift + click to select a range
756dfa9
one test
GregoryTravis Jan 30, 2024
58da2fd
hack
GregoryTravis Jan 31, 2024
f2cd94a
Merge branch 'develop' into wip/gmt/8578-Table.replace
GregoryTravis Jan 31, 2024
0501d05
revert hack
GregoryTravis Jan 31, 2024
fcab4e0
use merge
GregoryTravis Jan 31, 2024
f2a58f6
unhack
GregoryTravis Jan 31, 2024
5ed3fed
example
GregoryTravis Jan 31, 2024
8caac00
tests
GregoryTravis Jan 31, 2024
74bc949
tests
GregoryTravis Jan 31, 2024
69753fa
scramble lookup table order in tests
GregoryTravis Jan 31, 2024
4cf1309
duplicate inputs
GregoryTravis Jan 31, 2024
aae2bd2
self-lookup
GregoryTravis Jan 31, 2024
d458ed8
type test
GregoryTravis Jan 31, 2024
f924690
remove incorrect test
GregoryTravis Feb 1, 2024
bf142c0
remove materialize
GregoryTravis Feb 1, 2024
b97841c
db stub
GregoryTravis Feb 1, 2024
b88633e
unused imports, widgets
GregoryTravis Feb 1, 2024
2a02eb0
cleanup
GregoryTravis Feb 1, 2024
2b7ad13
Merge branch 'develop' into wip/gmt/8578-Table.replace
GregoryTravis Feb 1, 2024
8b7c361
rename replace_column to column
GregoryTravis Feb 1, 2024
9a084d0
docs, comment, cleanup
GregoryTravis Feb 1, 2024
64d43f3
comment
GregoryTravis Feb 1, 2024
4acb007
fix docs
GregoryTravis Feb 1, 2024
df35418
convert from Map
GregoryTravis Feb 1, 2024
5ac57ce
changelog
GregoryTravis Feb 1, 2024
d01152d
cleanup
GregoryTravis Feb 1, 2024
69cdb47
Merge branch 'develop' into wip/gmt/8578-Table.replace
GregoryTravis Feb 2, 2024
76368e7
review
GregoryTravis Feb 2, 2024
4ea9c22
review
GregoryTravis Feb 2, 2024
a269166
db except map
GregoryTravis Feb 2, 2024
b6cb8cc
wip
GregoryTravis Feb 2, 2024
940bfd3
move implementation to Replace_Helpers
GregoryTravis Feb 2, 2024
bb7cb2f
make_table_from_map method
GregoryTravis Feb 2, 2024
6c1d530
wip
GregoryTravis Feb 2, 2024
a475905
wip
GregoryTravis Feb 2, 2024
f9aaf71
review
GregoryTravis Feb 5, 2024
7ad9723
Merge branch 'develop' into wip/gmt/8578-Table.replace
GregoryTravis Feb 5, 2024
fe389f8
review
GregoryTravis Feb 5, 2024
645805b
update docs
GregoryTravis Feb 5, 2024
5450561
merge
GregoryTravis Feb 5, 2024
a6523c0
one col
GregoryTravis Feb 5, 2024
4dec328
wip
GregoryTravis Feb 5, 2024
d56bb09
two columns
GregoryTravis Feb 5, 2024
947ebcf
tests pass
GregoryTravis Feb 5, 2024
8ad298d
vector.transpose
GregoryTravis Feb 5, 2024
e9c1a2c
cleanup
GregoryTravis Feb 5, 2024
324cbef
wip
GregoryTravis Feb 5, 2024
7a5854c
parser error
GregoryTravis Feb 6, 2024
02a0d54
Merge branch 'develop' into wip/gmt/8578-Table.replace
GregoryTravis Feb 6, 2024
de5f91e
Merge branch 'wip/gmt/8578-Table.replace' into wip/gmt/8578-Table.rep…
GregoryTravis Feb 6, 2024
ea2876d
fix parser failure
GregoryTravis Feb 6, 2024
6fff368
wip
GregoryTravis Feb 6, 2024
80dbc76
Merge branch 'develop' into wip/gmt/8578-Table.replace
GregoryTravis Feb 6, 2024
8afbb74
Merge branch 'wip/gmt/8578-Table.replace' into wip/gmt/8578-Table.rep…
GregoryTravis Feb 6, 2024
e74b67d
wip
GregoryTravis Feb 6, 2024
4b34631
one test passes
GregoryTravis Feb 6, 2024
b241fbe
more tests
GregoryTravis Feb 6, 2024
21b6b80
wip
GregoryTravis Feb 6, 2024
fd9212a
max size tests
GregoryTravis Feb 6, 2024
fa7c024
from map tests
GregoryTravis Feb 6, 2024
fbea5a9
docs
GregoryTravis Feb 6, 2024
0073e46
Literal_Values
GregoryTravis Feb 6, 2024
a5bf58a
cleanup
GregoryTravis Feb 6, 2024
5a1b6d0
no table_builder
GregoryTravis Feb 6, 2024
ebf5087
Merge branch 'wip/gmt/8578-Table.replace' into wip/gmt/8578-Table.rep…
GregoryTravis Feb 6, 2024
cdddcc8
merge
GregoryTravis Feb 7, 2024
76ff386
wip
GregoryTravis Feb 7, 2024
22a76bb
wip
GregoryTravis Feb 7, 2024
0090a20
changelog
GregoryTravis Feb 7, 2024
b99627c
merge
GregoryTravis Feb 8, 2024
0028687
use proxy
GregoryTravis Feb 8, 2024
5eaf502
Merge branch 'develop' into wip/gmt/8578-Table.replace-db
GregoryTravis Feb 9, 2024
5a772f0
better error for length mismatch
GregoryTravis Feb 9, 2024
2c47cfe
wip
GregoryTravis Feb 9, 2024
32452e8
move transpose into Vector
GregoryTravis Feb 9, 2024
9bce18d
vector/array docs match
GregoryTravis Feb 9, 2024
bd3a300
merge
GregoryTravis Feb 12, 2024
4174e9b
warning on empty lookup table, tests
GregoryTravis Feb 12, 2024
ae66716
merge
GregoryTravis Feb 12, 2024
a1558b7
edge cases in empty lookup table
GregoryTravis Feb 12, 2024
3aeb792
merge
GregoryTravis Feb 13, 2024
5c0a54a
enable empty-table tests for db backend
GregoryTravis Feb 13, 2024
6af697b
review
GregoryTravis Feb 14, 2024
12a4c8e
Merge branch 'develop' into wip/gmt/8578-Table.replace-db
GregoryTravis Feb 14, 2024
88e022f
merge
GregoryTravis Feb 14, 2024
cd3fab6
no widgets for from/to columns
GregoryTravis Feb 16, 2024
046488c
use parameter name, not variable name
GregoryTravis Feb 16, 2024
2a7d8ae
Merge branch 'develop' into wip/gmt/8578-Table.replace-db
GregoryTravis Feb 16, 2024
fe72e3b
merge
GregoryTravis Feb 20, 2024
ea47095
fix merge
GregoryTravis Feb 20, 2024
7df7499
Merge branch 'develop' into wip/gmt/8578-Table.replace-db
GregoryTravis Feb 26, 2024
7cd0bb8
Merge branch 'develop' into wip/gmt/8578-Table.replace-db
GregoryTravis Feb 29, 2024
6a4ad4b
move implementation to Array_Like_Helpers
GregoryTravis Feb 29, 2024
72f36c7
fix tests
GregoryTravis Feb 29, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -615,6 +615,7 @@
- [Allow removing rows using a Filter_Condition.][8861]
- [Added `Table.to_xml`.][8979]
- [Implemented Write support for `S3_File`.][8921]
- [Implemented `Table.replace` for the database backend.][8986]
- [Separate `Group_By` from `columns` into new argument on `aggregate`.][9027]
- [Allow `copy_to` and `move_to` to work between local and S3 files.][9054]
- [Adjusted expression handling and new `Simple_Expression` type.][9128]
Expand Down Expand Up @@ -887,8 +888,9 @@
[8865]: https://github.com/enso-org/enso/pull/8865
[8935]: https://github.com/enso-org/enso/pull/8935
[8861]: https://github.com/enso-org/enso/pull/8861
[8979]: https://github.com/enso-org/enso/pull/8979
[8921]: https://github.com/enso-org/enso/pull/8921
[8979]: https://github.com/enso-org/enso/pull/8979
[8986]: https://github.com/enso-org/enso/pull/8986
[9027]: https://github.com/enso-org/enso/pull/9027
[9054]: https://github.com/enso-org/enso/pull/9054
[9128]: https://github.com/enso-org/enso/pull/9128
Expand Down
36 changes: 36 additions & 0 deletions distribution/lib/Standard/Base/0.0.0-dev/src/Data/Array.enso
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ import project.Errors.Common.Incomparable_Values
import project.Errors.Common.Index_Out_Of_Bounds
import project.Errors.Common.Not_Found
import project.Errors.Empty_Error.Empty_Error
import project.Errors.Illegal_Argument.Illegal_Argument
import project.Errors.Problem_Behavior.Problem_Behavior
import project.Errors.Unimplemented.Unimplemented
import project.Internal.Array_Like_Helpers
Expand Down Expand Up @@ -573,6 +574,41 @@ type Array
partition_with_index self predicate =
Array_Like_Helpers.partition_with_index self predicate

## GROUP Selections
Swaps the rows and columns of a matrix represented by an array of arrays.

! Error Conditions

- If the rows (subarrays) do not all have the same length, an
`Illegal_Argument` error is raised.

> Example
Transpose an array of arrays.

matrix = [[0, 1, 2].to_array, [3, 4, 5].to_array, [6, 7, 8].to_array].to_array
# +---+---+---+
# | 0 | 1 | 2 |
# +---+---+---+
# | 3 | 4 | 5 |
# +---+---+---+
# | 6 | 7 | 8 |
# +---+---+---+

transposed = [[0, 3, 6].to_array, [1, 4, 7].to_array, [2, 5, 8].to_array].to_array
# +---+---+---+
# | 0 | 3 | 6 |
# +---+---+---+
# | 1 | 4 | 7 |
# +---+---+---+
# | 2 | 5 | 8 |
# +---+---+---+

matrix.transposed == transposed
# => True
transpose : Array (Array Any) ! Illegal_Argument
transpose self =
Array_Like_Helpers.transpose self

## Applies a function to each element of the array, returning the `Vector`
of results.

Expand Down
35 changes: 35 additions & 0 deletions distribution/lib/Standard/Base/0.0.0-dev/src/Data/Vector.enso
Original file line number Diff line number Diff line change
Expand Up @@ -534,6 +534,41 @@ type Vector a
partition_with_index self predicate =
Array_Like_Helpers.partition_with_index self predicate

## GROUP Selections
Swaps the rows and columns of a matrix represented by a vector of vectors.

! Error Conditions

- If the rows (subvectors) do not all have the same length, an
`Illegal_Argument` error is raised.

> Example
Transpose a vector of vectors.

matrix = [[0, 1, 2], [3, 4, 5], [6, 7, 8]]
# +---+---+---+
# | 0 | 1 | 2 |
# +---+---+---+
# | 3 | 4 | 5 |
# +---+---+---+
# | 6 | 7 | 8 |
# +---+---+---+

transposed = [[0, 3, 6], [1, 4, 7], [2, 5, 8]]
# +---+---+---+
# | 0 | 3 | 6 |
# +---+---+---+
# | 1 | 4 | 7 |
# +---+---+---+
# | 2 | 5 | 8 |
# +---+---+---+

matrix.transposed == transposed
# => True
transpose : Vector (Vector Any) ! Illegal_Argument
transpose self =
Array_Like_Helpers.transpose self

## ICON dataframe_map_column
Applies a function to each element of the vector, returning the `Vector` of
results.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ private

import project.Any.Any
import project.Data.Array.Array
import project.Data.Array_Proxy.Array_Proxy
import project.Data.List.List
import project.Data.Map.Map
import project.Data.Maybe.Maybe
Expand Down Expand Up @@ -230,6 +231,15 @@ find vector condition start ~if_missing =
found = used_start.up_to self_len . find (idx -> (predicate (vector.at idx)))
if found.is_nothing then if_missing else vector.at found

transpose vec_of_vecs =
if vec_of_vecs.is_empty then [] else
length = vec_of_vecs.length
first_subvector_length = vec_of_vecs.at 0 . length
check_same_length vec_of_vecs <|
inner i = Vector.from_polyglot_array (Array_Proxy.new length j-> ((vec_of_vecs.at j).at i))
proxy = Array_Proxy.new first_subvector_length inner
Vector.from_polyglot_array proxy

map vector function on_problems =
vector_from_function vector.length (function << vector.at) on_problems

Expand Down Expand Up @@ -323,3 +333,18 @@ filter_with_index vector predicate =
builder = vector.fold_with_index Vector.new_builder builder-> ix-> elem->
if predicate ix elem then builder.append elem else builder
builder.to_vector

## PRIVATE
Check that all vectors have the same length and return an informative message
if they don't.

Compares all vectors to the first one and reports the first one that differs.
check_same_length : Vector (Vector Any) -> Any -> Any ! Illegal_Argument
check_same_length vecs ~action =
if vecs.is_empty then action else
num_vecs = vecs.length
len = vecs.at 0 . length
go i = if i >= num_vecs then action else
if vecs.at i . length == len then @Tail_Call go (i+1) else
Error.throw (Illegal_Argument.Error "Transpose requires that all vectors be the same length, but rows 0 and "+i.to_text+" had different lengths ("+len.to_text+" and "+(vecs.at i . length).to_text+")")
go 0
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ import Standard.Table.Internal.Aggregate_Column_Helper
import Standard.Table.Internal.Column_Naming_Helper.Column_Naming_Helper
import Standard.Table.Internal.Constant_Column.Constant_Column
import Standard.Table.Internal.Problem_Builder.Problem_Builder
import Standard.Table.Internal.Replace_Helpers
import Standard.Table.Internal.Table_Helpers
import Standard.Table.Internal.Table_Helpers.Table_Column_Helper
import Standard.Table.Internal.Unique_Name_Strategy.Unique_Name_Strategy
Expand Down Expand Up @@ -935,6 +936,74 @@ type DB_Table
on_problems.attach_problems_before problems <|
Warning.set result []

## PRIVATE
A helper that creates a two-column table from a Map.

The keys of the `Map` become the first column, with name
`key_column_name`, and the values of the `Map` become the second column,
with name `value_column_name`.

For the in-memory database, the `Map` can be empty. For the database
backends, it must not be empty.

Arguments:
- map: The `Map` to create the table from.
- key_column_name: The name to use for the first column.
- value_column_name: The name to use for the second column.
make_table_from_map : Map Any Any -> Text -> Text -> Table
make_table_from_map self map key_column_name value_column_name =
total_size = map.size * 2

if map.is_empty then Error.throw (Illegal_Argument.Error "Map argument cannot be empty") else
if total_size > MAX_LITERAL_ELEMENT_COUNT then Error.throw (Illegal_Argument.Error "Map argument is too large ("+map.size.to_text+" entries): materialize a table into the database instead") else
keys_and_values = map.to_vector
self.make_table_from_vectors [keys_and_values.map .first, keys_and_values.map .second] [key_column_name, value_column_name]

## PRIVATE
A helper that creates a literal table from `Vector`s.

For the in-memory database, the columns can be empty. For the database
backends, they must not be empty.

Arguments:
- column_vectors: A `Vector` of `Vector`s; each inner `Vector` becomes a
column of the table.
- column_names: The names of the columns of the new table.
make_table_from_vectors : Vector (Vector Any) -> Vector Text -> DB_Table
make_table_from_vectors self column_vectors column_names =
Runtime.assert (column_vectors.length == column_names.length) "column_vectors and column_names must have the same length"

# Assume the columns are all the same length; if not, it will be an error anyway.
total_size = if column_vectors.is_empty || column_vectors.at 0 . is_empty then 0 else
column_vectors.length * (column_vectors.at 0 . length)

if total_size == 0 then Error.throw (Illegal_Argument.Error "Vectors cannot be empty") else
if total_size > MAX_LITERAL_ELEMENT_COUNT then Error.throw (Illegal_Argument.Error "Too many elements for table literal ("+total_size.to_text+"): materialize a table into the database instead") else
type_mapping = self.connection.dialect.get_type_mapping

values_to_type_ref column_vector =
value_type = Value_Type_Helpers.find_common_type_for_arguments column_vector
sql_type = case value_type of
Nothing -> SQL_Type.null
_ -> type_mapping.value_type_to_sql value_type Problem_Behavior.Ignore
SQL_Type_Reference.from_constant sql_type

literal_table_name = self.connection.base_connection.table_naming_helper.generate_random_table_name "enso-literal-"

from_spec = From_Spec.Literal_Values column_vectors column_names literal_table_name
context = Context.for_subquery from_spec

internal_columns = 0.up_to column_vectors.length . map i->
column_vector = column_vectors.at i
column_name = column_names.at i

type_ref = values_to_type_ref column_vector.to_vector
generated_literal_column_name = "column"+(i+1).to_text
sql_expression = SQL_Expression.Column literal_table_name generated_literal_column_name
Internal_Column.Value column_name type_ref sql_expression

DB_Table.Value literal_table_name self.connection internal_columns context

## PRIVATE

Create a constant column from a value.
Expand Down Expand Up @@ -1404,7 +1473,7 @@ type DB_Table
In the Database backend, there are no guarantees related to ordering of
results.

? Error Conditions
! Error Conditions

- If this table or the lookup table is lacking any of the columns
specified in `key_columns`, a `Missing_Input_Columns` error is raised.
Expand Down Expand Up @@ -1460,7 +1529,7 @@ type DB_Table
In the Database backend, there are no guarantees related to ordering of
results.

? Error Conditions
! Error Conditions

- If this table or the lookup table is lacking any of the columns
specified by `from_column`, `to_column`, or `column`, a
Expand Down Expand Up @@ -1506,10 +1575,10 @@ type DB_Table
# 1 | 20 | b | f
# 2 | 30 | c | g
# 3 | 40 | d | h
@column Widget_Helpers.make_column_name_selector
replace : DB_Table | Map -> (Text | Integer) -> (Text | Integer) -> (Text | Integer) -> Boolean -> Problem_Behavior -> DB_Table ! Missing_Input_Columns | Non_Unique_Key | Unmatched_Rows_In_Lookup
replace self lookup_table:(DB_Table | Map) column:(Text | Integer) from_column:(Text | Integer)=0 to_column:(Text | Integer)=1 allow_unmatched_rows:Boolean=True on_problems:Problem_Behavior=Problem_Behavior.Report_Warning =
_ = [lookup_table, column, from_column, to_column, allow_unmatched_rows, on_problems]
Error.throw (Unsupported_Database_Operation.Error "DB_Table.replace is not implemented yet for the Database backends.")
replace self lookup_table:(DB_Table | Map) column:(Text | Integer) from_column:(Text | Integer | Nothing)=Nothing to_column:(Text | Integer | Nothing)=Nothing allow_unmatched_rows:Boolean=True on_problems:Problem_Behavior=Problem_Behavior.Report_Warning =
Replace_Helpers.replace self lookup_table column from_column to_column allow_unmatched_rows on_problems

## ALIAS join by row position
GROUP Standard.Base.Calculations
Expand Down Expand Up @@ -2809,3 +2878,7 @@ DB_Table.from (that:Table) =

## PRIVATE
Table_Ref.from (that:DB_Table) = Table_Ref.Value that

## PRIVATE
The largest dataset that can be used to make a literal table, expressed in number of elements.
MAX_LITERAL_ELEMENT_COUNT = 256
Original file line number Diff line number Diff line change
Expand Up @@ -341,6 +341,10 @@ generate_from_part dialect from_spec = case from_spec of
dialect.wrap_identifier name ++ alias dialect as_name
From_Spec.Query raw_sql as_name ->
SQL_Builder.code raw_sql . paren ++ alias dialect as_name
From_Spec.Literal_Values vecs column_names as_name ->
Runtime.assert (vecs.length == column_names.length) "Vectors and column names must have the same length"
values = SQL_Builder.join ", " (vecs.transpose.map (vec-> SQL_Builder.join ", " (vec.map SQL_Builder.interpolation) . paren))
SQL_Builder.code "(VALUES " ++ values ++ ")" ++ alias dialect as_name
From_Spec.Join kind left_spec right_spec on ->
left = generate_from_part dialect left_spec
right = generate_from_part dialect right_spec
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,17 @@ type From_Spec
the same table.
Query (raw_sql : Text) (alias : Text)

## PRIVATE

A query source consisting of a literal VALUES clause.

Arguments:
- column_vectors: the contents of the literal table's columns.
- column_names: the names of the literal table's columns,
- alias: the name by which the table can be referred to in other parts of
the query.
Literal_Values (column_vectors : Vector (Vector Any)) (column_names : Vector Text) (alias : Text)
GregoryTravis marked this conversation as resolved.
Show resolved Hide resolved

## PRIVATE

A query source that performs a join operation on two sources.
Expand Down
Loading
Loading