Feat - add custom filter parser by kodumbeats · Pull Request #3 · utopia-php/database

kodumbeats · 2021-04-22T18:06:43Z

New Feature

This PR introduces a new Query class which will hold all related information for custom document filters.

Note - due to $filter being used already, I refer to a custom filter as a $query until we can find something more clear.

Test plan

Tests included with PR

Todo

Implement filter parser with no nesting
Create new Query class
Add tests
Unit test parseExpression()

Should we...

Check types against collection schema
Expand tests to check all query types

kodumbeats · 2021-04-22T18:13:00Z

Testing locally against MariaDB has 🟢

$ docker-compose exec tests vendor/bin/phpunit --debug /usr/src/code/tests/Database/Adapter/MariaDBTest.php

PHPUnit 9.5.4 by Sebastian Bergmann and contributors.

Test 'Utopia\Tests\Adapter\MariaDBTest::testCreateDelete' started
Test 'Utopia\Tests\Adapter\MariaDBTest::testCreateDelete' ended
Test 'Utopia\Tests\Adapter\MariaDBTest::testCreateDeleteCollection' started
Test 'Utopia\Tests\Adapter\MariaDBTest::testCreateDeleteCollection' ended
Test 'Utopia\Tests\Adapter\MariaDBTest::testCreateDeleteAttribute' started
Test 'Utopia\Tests\Adapter\MariaDBTest::testCreateDeleteAttribute' ended
Test 'Utopia\Tests\Adapter\MariaDBTest::testCreateDeleteIndex' started
Test 'Utopia\Tests\Adapter\MariaDBTest::testCreateDeleteIndex' ended
Test 'Utopia\Tests\Adapter\MariaDBTest::testCreateDocument' started
Test 'Utopia\Tests\Adapter\MariaDBTest::testCreateDocument' ended
Test 'Utopia\Tests\Adapter\MariaDBTest::testGetDocument' started
Test 'Utopia\Tests\Adapter\MariaDBTest::testGetDocument' ended
Test 'Utopia\Tests\Adapter\MariaDBTest::testUpdateDocument' started
Test 'Utopia\Tests\Adapter\MariaDBTest::testUpdateDocument' ended
Test 'Utopia\Tests\Adapter\MariaDBTest::testDeleteDocument' started
Test 'Utopia\Tests\Adapter\MariaDBTest::testDeleteDocument' ended
Test 'Utopia\Tests\Adapter\MariaDBTest::testFindFirst' started
Test 'Utopia\Tests\Adapter\MariaDBTest::testFindFirst' ended
Test 'Utopia\Tests\Adapter\MariaDBTest::testFindLast' started
Test 'Utopia\Tests\Adapter\MariaDBTest::testFindLast' ended
Test 'Utopia\Tests\Adapter\MariaDBTest::testParseQueries' started
Test 'Utopia\Tests\Adapter\MariaDBTest::testParseQueries' ended


Time: 00:03.764, Memory: 6.00 MB

OK (11 tests, 101 assertions)

eldadfux · 2021-04-23T05:38:42Z

Thinking about this a bit here are some of my ideas:

We will create a class called Database\Query to hold all query related operations/data
The class constructor will create a query instance with all relevant metadata that will allow us to build a native SQL/NOSQL query (key, operator, value/values), an array of query objects will allow us to build complex queries.
Create a static method called Database\Query::parse([STRING]) that will parse a string query (director.equals('brandon')) and return a Database\Query instance.
The Database\Query class can have multiple getters and setters to make DX nicer.

kodumbeats · 2021-04-23T16:32:00Z

As of e802153

# kodumbeats @ magician in ~/w/kodumbeats/database on git:v0 x [12:28:00]
$ dc exec tests vendor/bin/phpunit tests/Database/Adapter/MariaDBTest.php 
PHPUnit 9.5.4 by Sebastian Bergmann and contributors.

..........                                                        10 / 10 (100%)

Time: 00:00.711, Memory: 6.00 MB

OK (10 tests, 85 assertions)

# kodumbeats @ magician in ~/w/kodumbeats/database on git:v0 x [12:28:17] 
$ dc exec tests vendor/bin/phpunit tests/Database/QueryTest.php          
PHPUnit 9.5.4 by Sebastian Bergmann and contributors.

..                                                                  2 / 2 (100%)

Time: 00:00.002, Memory: 6.00 MB

OK (2 tests, 10 assertions)

eldadfux · 2021-04-23T16:44:26Z

+        return [$operator, $operand];
+    }
+
+    // /**


Love this idea! we should probably have a validator class for queries syntax...

How's this for a start? I figure this is a good place to check if the operator is supported too:
179a0d5

Looks like a good start, we can have it as a standalone PR

This reverts commit f581824.

This reverts commit 1c6e60c.

eldadfux · 2021-04-25T15:32:12Z

We also need to support multiple params for or queries, a few examples:

director.equals('eldad', 'brandon')
director.equals('eldad','brandon')

kodumbeats · 2021-04-26T15:53:56Z

As of 631ecfd

$ docker-compose exec tests vendor/bin/phpunit --debug tests/Database/QueryTest.php
PHPUnit 9.5.4 by Sebastian Bergmann and contributors.

Test 'Utopia\Tests\QueryTest::testParse' started
Test 'Utopia\Tests\QueryTest::testParse' ended
Test 'Utopia\Tests\QueryTest::testGetAttribute' started
Test 'Utopia\Tests\QueryTest::testGetAttribute' ended
Test 'Utopia\Tests\QueryTest::testGetOperator' started
Test 'Utopia\Tests\QueryTest::testGetOperator' ended
Test 'Utopia\Tests\QueryTest::testGetValue' started
Test 'Utopia\Tests\QueryTest::testGetValue' ended
Test 'Utopia\Tests\QueryTest::testGetQuery' started
Test 'Utopia\Tests\QueryTest::testGetQuery' ended


Time: 00:00.002, Memory: 6.00 MB

OK (5 tests, 22 assertions)

eldadfux · 2021-04-26T16:23:28Z

Looks good! I would have added more tests to make sure casting works as expected with all types

eldadfux · 2021-04-26T19:00:34Z

+                return null;
+            } else {
+                // strip quotes from queries of type string
+                return str_replace(['"',"'"], "", $value);


We need to remove only the first and last chars if they are " or '

Maybe we should also handle escaping of \' and \"

Used trim() and stripslashes() for this

#1 Drop $enable flag on skipDuplicates() scope guard The $enable param made every non-skipDuplicates createDocuments call pay for a closure allocation + extra function call. Branch at the call site instead so the cost only applies when the flag is actually set. - Adapter::skipDuplicates(callable, bool) → skipDuplicates(callable) - Database::skipDuplicates(callable, bool) → skipDuplicates(callable) - Database::createDocuments, Mirror::createDocuments, Pool::delegate, Pool::withTransaction now branch inline. #2 Drop fetchExistingByIds helper, inline find() The helper's per-tenant grouping defended a hypothetical multi-tenant batching scenario that no caller exercises (relationships are intra- tenant, callers always batch per tenant). Existing patterns in the same file (refetchDocuments, relationship loading) just call find() directly. Match that idiom and drop ~70 lines. #4 Mirror: only capture inserted docs in skipDuplicates mode The captureOnNext accumulator paid the cost (closure + per-doc array push) on every createDocuments call, including the common non-skip path. Branch at the entry of Mirror::createDocuments so the capture only happens when skipDuplicates is set; the non-skip path passes through to source/destination unchanged. #5 Move getInsertKeyword/Suffix/PermissionsSuffix to getters cluster Were sitting next to createDocuments(); moved to the getSupport* cluster around line 1030 where other adapter-capability shims live. Not addressed: - #2 partial: the existing patterns (refetchDocuments etc.) don't handle tenant-per-document multi-tenant batches either, so this is consistent. - #3 (drop the pre-filter): rejected. createDocumentRelationships runs in the encoding loop BEFORE the adapter's INSERT IGNORE no-ops the parent, so dropping the pre-filter would deterministically duplicate child rows on every CSV re-import of a collection with relationships (not a race window — every call). The relationships test verifies this. Reverting would require reintroducing the deferred-relationships scaffolding we just removed, and the adapter still couldn't tell us which parents were actually inserted (SQL INSERT IGNORE has no per-row reporting). Pre-filter stays.

kodumbeats added 3 commits April 22, 2021 14:00

Add function to parse query expressions

3cf58ce

Add method to parse custom filter queries

514bf70

Add tests for parsing queries

80eec64

kodumbeats marked this pull request as draft April 22, 2021 18:09

Add test for parsing query expressions

69e6bc7

kodumbeats added 9 commits April 23, 2021 11:21

Refactor query into standalone class

cdf5267

Add getters for object properties

3f75885

Add method to get all query details

2a8a5b3

Add return typing

f763ba4

Clean up code

06f02f7

Call parseExpression as a static method

e01fcc4

Add tests for query class

65a2166

Refactor parsing code into query class and tests

1bb848d

Add type hinting

e802153

Merge branch 'v0' into v0

83a09a3

kodumbeats marked this pull request as ready for review April 23, 2021 16:37

eldadfux requested changes Apr 23, 2021

View reviewed changes

kodumbeats added 10 commits April 23, 2021 12:48

Add types to constructor params

c18b485

Rename operand to value

b2f6603

Change to protected method

f581824

Use bracket syntax for switch

fe6becb

Treat query expression as kv pair

1c6e60c

Create query validator

179a0d5

Remove references to old var

8ada10e

Revert "Change to protected method"

8934ce7

This reverts commit f581824.

Revert "Treat query expression as kv pair"

38b02cf

This reverts commit 1c6e60c.

Use value instead of operand

c2b9dc2

kodumbeats added 6 commits April 26, 2021 11:41

Typecast expression values and handle comma-separated values

6b3bcd2

Test with integer param

a1b472d

Test for typecasted bool expressions

e74da85

Test multiple query values

f7e86b6

Cleanup for readability

a14a730

Reserve query validator for separate PR

631ecfd

kodumbeats added 3 commits April 26, 2021 13:53

Only consider periods oudside parentheses

4d6189b

Test for float values

1f4c64f

Handle null as value

fa28286

eldadfux requested changes Apr 26, 2021

View reviewed changes

kodumbeats added 5 commits April 26, 2021 15:21

Refactor elseifs into switch

2ec2fc8

Use single array_map to trim whitespace and typecast

ab2a6b7

Remove unneeded switch case

b01d9dd

Trim whitespace and remove escape slashes

7db5c8f

Test for whitespace and escaped character

40cd42c

TorstenDittmann reviewed Apr 26, 2021

View reviewed changes

Comment thread src/Database/Query.php Outdated

Comment thread src/Database/Query.php Outdated

kodumbeats added 3 commits April 26, 2021 16:29

Use strrpos to find last occurrence of parenthesis

53cf8dc

Correct comments

ba852c2

Test constructor function

87dd5b3

eldadfux requested changes Apr 26, 2021

View reviewed changes

Comment thread tests/Database/QueryTest.php Outdated

Properly handle whitespace between values

1522284

eldadfux merged commit 60391ec into utopia-php:v0 Apr 26, 2021

coderabbitai Bot mentioned this pull request Aug 8, 2025

Implement find method caching #656

Closed

coderabbitai Bot mentioned this pull request Oct 17, 2025

Add vector attribute support for PostgreSQL with pgvector extension #665

Merged

coderabbitai Bot mentioned this pull request Dec 2, 2025

Joins #763

Open

coderabbitai Bot mentioned this pull request Dec 18, 2025

Handle non utf chars #773

Merged

Conversation

kodumbeats commented Apr 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

New Feature

Test plan

Todo

Should we...

Uh oh!

kodumbeats commented Apr 22, 2021

Uh oh!

eldadfux commented Apr 23, 2021

Uh oh!

kodumbeats commented Apr 23, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

eldadfux Apr 23, 2021

Choose a reason for hiding this comment

Uh oh!

kodumbeats Apr 23, 2021

Choose a reason for hiding this comment

Uh oh!

eldadfux Apr 23, 2021

Choose a reason for hiding this comment

Uh oh!

eldadfux commented Apr 25, 2021

Uh oh!

kodumbeats commented Apr 26, 2021

Uh oh!

eldadfux commented Apr 26, 2021

Uh oh!

Uh oh!

eldadfux Apr 26, 2021

Choose a reason for hiding this comment

Uh oh!

eldadfux Apr 26, 2021

Choose a reason for hiding this comment

Uh oh!

kodumbeats Apr 26, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kodumbeats commented Apr 22, 2021 •

edited

Loading