Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Je/query index stage 1 #6715

Merged
merged 4 commits into from
Jun 19, 2023
Merged
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -3,6 +3,8 @@
### Enhancements
* <New feature description> (PR [#????](https://github.com/realm/realm-core/pull/????))
* Storage of Decimal128 properties has been optimised so that the individual values will take up 0 bits (if all nulls), 32 bits, 64 bits or 128 bits depending on what is needed. (PR [#6111]https://github.com/realm/realm-core/pull/6111))
* You can have a collection embedded in any Mixed property (except Set<Mixed>).
* Quering a specific entry in a list-of-primitives (in particular 'first and 'last') is supported.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍


### Fixed
* <How do the end-user experience this issue? what was the impact?> ([#????](https://github.com/realm/realm-core/issues/????), since v?.?.?)
44 changes: 44 additions & 0 deletions src/realm/collection.cpp
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
#include <realm/collection.hpp>
#include <realm/bplustree.hpp>
#include <realm/array_key.hpp>
#include <realm/array_string.hpp>
#include <realm/array_mixed.hpp>

namespace realm {

@@ -91,6 +93,48 @@ void check_for_last_unresolved(BPlusTree<ObjKey>* tree)

Collection::~Collection() {}

Mixed Collection::get_any(Mixed val, Path::const_iterator begin, Path::const_iterator end, Allocator& alloc)
{
auto path_size = end - begin;
if (val.is_type(type_Dictionary) && begin->is_key()) {
Array top(alloc);
top.init_from_ref(val.get_ref());

BPlusTree<StringData> keys(alloc);
keys.set_parent(&top, 0);
keys.init_from_parent();
size_t ndx = keys.find_first(StringData(begin->get_key()));
if (ndx != realm::not_found) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to see some tests where the key is found and there is more levels in the path, but the dictionary key contains a simple primitive value and not another nested level. In that case, we should return Null early and not recurse further (also for lists).

Copy link
Contributor Author

@jedelbo jedelbo Jun 16, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will add some tests. We will always make a recursive call if the key is found, but that call will return quickly as non of the tests for type will succeed. Do you think this is a problem? Otherwise we would have to do the tests twice.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That seems fine to me 👍

BPlusTree<Mixed> values(alloc);
values.set_parent(&top, 1);
values.init_from_parent();
val = values.get(ndx);
if (path_size > 1) {
val = Collection::get_any(val, begin + 1, end, alloc);
}
return val;
}
}
if (val.is_type(type_List) && begin->is_ndx()) {
ArrayMixed list(alloc);
list.init_from_ref(val.get_ref());
if (size_t sz = list.size()) {
auto idx = begin->get_ndx();
if (idx == size_t(-1)) {
idx = sz - 1;
}
if (idx < sz) {
val = list.get(idx);
}
if (path_size > 1) {
val = Collection::get_any(val, begin + 1, end, alloc);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be a problem (probably we are never going to hit it though), in case of nested collections that are too deep, we might run out of stack space.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any clue when that would be a problem?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is up to how long is the nested collection... List<List<List<List..... will result in N recursive call proportional to the amount of collections that are nested. Most probably we will never run into this problem in practise, and there are also others part in the code when this can manifest earlier than at this point.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a check that nesting level will not exceed 100.

}
return val;
}
}
return {};
}

std::pair<std::string, std::string> CollectionBase::get_open_close_strings(size_t link_depth,
JSONOutputMode output_mode) const
{
2 changes: 2 additions & 0 deletions src/realm/collection.hpp
Original file line number Diff line number Diff line change
@@ -94,6 +94,8 @@ class Collection {
virtual Path get_short_path() const = 0;
// Return a path based on keys instead of indices
virtual StablePath get_stable_path() const = 0;

static Mixed get_any(Mixed, Path::const_iterator, Path::const_iterator, Allocator&);
};

using CollectionPtr = std::shared_ptr<Collection>;
19 changes: 18 additions & 1 deletion src/realm/collection_parent.cpp
Original file line number Diff line number Diff line change
@@ -33,7 +33,16 @@ namespace realm {
std::ostream& operator<<(std::ostream& ostr, const PathElement& elem)
{
if (elem.is_ndx()) {
ostr << elem.get_ndx();
size_t ndx = elem.get_ndx();
if (ndx == 0) {
ostr << "FIRST";
}
else if (ndx == size_t(-1)) {
ostr << "LAST";
}
else {
ostr << elem.get_ndx();
}
}
else if (elem.is_col_key()) {
ostr << elem.get_col_key();
@@ -45,6 +54,14 @@ std::ostream& operator<<(std::ostream& ostr, const PathElement& elem)
return ostr;
}

std::ostream& operator<<(std::ostream& ostr, const Path& path)
{
for (auto& elem : path) {
ostr << '[' << elem << ']';
}
return ostr;
}

/***************************** CollectionParent ******************************/

CollectionParent::~CollectionParent() {}
5 changes: 3 additions & 2 deletions src/realm/collection_parent.hpp
Original file line number Diff line number Diff line change
@@ -199,10 +199,11 @@ struct PathElement {
}
};

std::ostream& operator<<(std::ostream& ostr, const PathElement& elem);

using Path = std::vector<PathElement>;

std::ostream& operator<<(std::ostream& ostr, const PathElement& elem);
std::ostream& operator<<(std::ostream& ostr, const Path& path);

// Path from the group level.
struct FullPath {
TableKey top_table;
44 changes: 38 additions & 6 deletions src/realm/parser/driver.cpp
Original file line number Diff line number Diff line change
@@ -537,7 +537,7 @@ Query EqualityNode::visit(ParserDriver* drv)
else if (right->has_single_value() && (left_type == right_type || left_type == type_Mixed)) {
Mixed val = right->get_mixed();
const ObjPropertyBase* prop = dynamic_cast<const ObjPropertyBase*>(left.get());
if (prop && !prop->links_exist()) {
if (prop && !prop->links_exist() && !prop->has_path()) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is only managing simple values, right? No links?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No links and no path.

auto col_key = prop->column_key();
if (val.is_null()) {
switch (op) {
@@ -848,17 +848,29 @@ std::unique_ptr<Subexpr> PropertyNode::visit(ParserDriver* drv, DataType)
}
std::unique_ptr<Subexpr> subexpr{drv->column(m_link_chain, path)};

if (!path->at_end()) {
if (path->is_identifier()) {
auto trailing = path->next_identifier();
Path indexes;
while (!path->at_end()) {
indexes.emplace_back(std::move(*(path->current_path_elem++)));
}

if (!indexes.empty()) {
const PathElement& first_index = indexes.front();
if (indexes.size() > 1 && subexpr->get_type() != type_Mixed) {
throw InvalidQueryError("Only Property of type 'any' can have nested collections");
}
if (auto mixed = dynamic_cast<Columns<Mixed>*>(subexpr.get())) {
mixed->path(indexes);
}
else if (first_index.is_key()) {
auto trailing = first_index.get_key();
if (auto dict = dynamic_cast<Columns<Dictionary>*>(subexpr.get())) {
if (trailing == "@values") {
}
else if (trailing == "@keys") {
subexpr = std::make_unique<ColumnDictionaryKeys>(*dict);
}
else {
dict->key(trailing);
dict->key(indexes);
}
}
else {
@@ -879,9 +891,29 @@ std::unique_ptr<Subexpr> PropertyNode::visit(ParserDriver* drv, DataType)
}
}
else {
throw InvalidQueryError("Index not supported");
// This must be an integer index
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we assert this somehow?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Certainly

auto ok = false;
if (indexes.size() == 1) {
if (auto coll = dynamic_cast<ColumnListBase*>(subexpr.get())) {
ok = coll->index(first_index);
}
}
else {
if (auto coll = dynamic_cast<Columns<Lst<Mixed>>*>(subexpr.get())) {
ok = coll->indexes(indexes);
}
}
if (!ok) {
throw InvalidQueryError(util::format("Property '%1.%2' does not support index '%3'",
m_link_chain.get_current_table()->get_class_name(), identifier,
first_index));
}
}
}
/*
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove... commented code

if (!path->path_elems.back().index.is_null()) {
}
*/
if (post_op) {
return post_op->visit(drv, subexpr.get());
}
8 changes: 0 additions & 8 deletions src/realm/parser/driver.hpp
Original file line number Diff line number Diff line change
@@ -284,18 +284,10 @@ class PathNode : public ParserNode {
{
return current_path_elem == path_elems.end();
}
bool is_identifier() const
{
return current_path_elem->is_key();
}
const std::string& next_identifier()
{
return (current_path_elem++)->get_key();
}
size_t next_index()
{
return (current_path_elem++)->get_ndx();
}
const std::string& last_identifier()
{
return path_elems.back().get_key();
Loading