Skip to content

eng-251#351

Merged
Connorbelez merged 2 commits intomainfrom
Connorbelez/eng-251-record-queries
Mar 30, 2026
Merged

eng-251#351
Connorbelez merged 2 commits intomainfrom
Connorbelez/eng-251-record-queries

Conversation

@Connorbelez
Copy link
Copy Markdown
Owner

@Connorbelez Connorbelez commented Mar 29, 2026

TL;DR

Implements comprehensive record querying and search functionality for the CRM system with EAV (Entity-Attribute-Value) field assembly, filtering, sorting, and full-text search capabilities.

What changed?

Added three new query functions to handle CRM record operations:

  • queryRecords: Retrieves records with optional filtering and sorting. Uses dual code paths - native Convex pagination for simple queries, and in-memory filtering/sorting for complex queries with a 500-record cap
  • getRecord: Fetches a single record with all field values assembled from EAV tables, plus inbound/outbound linked records via the recordLinks table
  • searchRecords: Performs full-text search on record labels using Convex's search index

The implementation includes:

  • EAV field assembly that queries only relevant value tables based on field types
  • Comprehensive filtering with operators like eq, gt, contains, is_any_of, etc.
  • Sorting by field values with ascending/descending direction
  • Parallel field value loading for performance optimization
  • Type definitions for unified record shapes, filters, and linked records
  • Search index on the records table's labelValue field

How to test?

  1. Run bun check and bun typecheck to verify type safety
  2. Execute bunx convex codegen to generate updated types
  3. Test the three query endpoints:
    • Call queryRecords with various filter/sort combinations
    • Use getRecord to fetch individual records with their links
    • Try searchRecords with different search terms

Why make this change?

This implements the core querying infrastructure needed for the CRM system to retrieve and display record data. The EAV model requires specialized assembly logic to reconstruct records from normalized value tables, while the dual pagination approach ensures both performance (for simple queries) and functionality (for complex filtered queries). The search capability enables users to quickly find records by their display labels.

Summary by CodeRabbit

Release Notes

  • New Features

    • Implemented record querying with filtering and sorting capabilities
    • Added ability to fetch individual records with relationship data (inbound and outbound links)
    • Introduced full-text search functionality for records by label value
    • Added pagination support for efficient record retrieval
  • Documentation

    • Added technical specifications and task definitions for record query implementation

@linear
Copy link
Copy Markdown

linear bot commented Mar 29, 2026

Copy link
Copy Markdown

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry @Connorbelez, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai bot commented Mar 29, 2026

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 41b5a6aa-e1a9-484f-87d4-aa1989219272

📥 Commits

Reviewing files that changed from the base of the PR and between bcf99d1 and 163526f.

📒 Files selected for processing (2)
  • convex/crm/recordQueries.ts
  • convex/crm/types.ts
✅ Files skipped from review due to trivial changes (2)
  • convex/crm/types.ts
  • convex/crm/recordQueries.ts

📝 Walkthrough

Walkthrough

The changes add three new Convex query handlers for CRM record management: queryRecords for paginated queries with filtering/sorting, getRecord for single record retrieval with link resolution, and searchRecords for full-text label search. Supporting types and a search index on labelValue enable these operations on EAV-modeled record data.

Changes

Cohort / File(s) Summary
CRM Query Handlers
convex/crm/recordQueries.ts
Implements three Convex query handlers with distinct strategies: queryRecords supports native pagination (no filters/sort) and offset pagination (with filters/sort), getRecord fetches single records and resolves outbound/inbound links, and searchRecords queries the search index. Includes helpers for reading EAV values from typed tables, assembling records from active field definitions, applying AND-combined filters with fail-closed semantics, and sorting.
CRM Type Definitions
convex/crm/types.ts
Exports shared types: UnifiedRecord (common return shape with fields map), RecordFilter (field-level filters with typed operators), RecordSort (field sorting direction), QueryRecordsResult (paginated results with cursors), GetRecordResult (record with links split into outbound/inbound), and LinkedRecord (link references with optional label).
Schema Updates
convex/schema.ts
Adds searchIndex("search_label") to records table on labelValue field with filters for orgId, objectDefId, and isDeleted to enable scoped full-text search.
Documentation
specs/ENG-251/tasks.md, specs/ENG-251/chunks/manifest.md
Adds task list and chunk manifest for ENG-251, documenting completed implementation steps T-001 through T-010 and final quality gates.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~30 minutes

Possibly related issues

  • Issue #329 — Directly implements Phase-2 CRM features (queryRecords, getRecord, searchRecords handlers with EAV-to-UnifiedRecord assembly and field definition loading).

Possibly related PRs

  • PR #350 — Introduces the CRM schema foundations (records, recordValues* tables, recordLinks, field-type mappings) that these query handlers depend on and operate against.
  • PR #340 — Provides the objectDefs and fieldDefs schema structures that are loaded and referenced throughout the query assembly logic.

Poem

🐰 Three queries hop through EAV arrays,
Assembling fields in countless ways,
With filters, sorts, and links combined—
The perfect records, signed and aligned!

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Title check ⚠️ Warning The title 'eng-251' is a ticket identifier and is not descriptive of the actual changes in the pull request. Rename the title to clearly describe the main functionality being added, such as 'Add CRM record queries and search with EAV field assembly' or 'Implement queryRecords, getRecord, and searchRecords query handlers'.
✅ Passed checks (2 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch Connorbelez/eng-251-record-queries

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown
Owner Author

This stack of pull requests is managed by Graphite. Learn more about stacking.

@Connorbelez Connorbelez marked this pull request as ready for review March 29, 2026 20:59
Copilot AI review requested due to automatic review settings March 29, 2026 20:59
Copy link
Copy Markdown

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your free trial has ended. If you'd like to continue receiving code reviews, you can add a payment method here.

Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Implements core CRM record retrieval for the EAV data model, including field assembly from typed value tables, filtering/sorting, record link hydration, and label full-text search via a new Convex search index.

Changes:

  • Added searchIndex("search_label") on records.labelValue to support full-text search with org/object/deleted filtering.
  • Introduced shared CRM query types (UnifiedRecord, filters, sorts, link shapes).
  • Added queryRecords, getRecord, and searchRecords queries with EAV fan-out field assembly and filtering/sorting logic.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 11 comments.

Show a summary per file
File Description
specs/ENG-251/tasks.md Task checklist for ENG-251 implementation work.
specs/ENG-251/chunks/manifest.md Chunking/rollup notes for ENG-251.
convex/schema.ts Adds search_label search index on records for label search with filter fields.
convex/crm/types.ts Adds CRM query/response types for unified record and linking shapes.
convex/crm/recordQueries.ts Implements record querying, field assembly, filtering/sorting, link hydration, and label search.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread convex/crm/recordQueries.ts Outdated
Comment thread convex/crm/recordQueries.ts
Comment thread convex/crm/recordQueries.ts
Comment thread convex/crm/recordQueries.ts Outdated
Comment thread convex/crm/recordQueries.ts
Comment thread convex/crm/recordQueries.ts Outdated
Comment thread convex/crm/recordQueries.ts
Comment thread convex/crm/recordQueries.ts Outdated
Comment thread convex/crm/recordQueries.ts Outdated
Comment thread convex/crm/recordQueries.ts Outdated
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (2)
convex/crm/recordQueries.ts (2)

270-275: Operator validation is loose compared to the type definition.

The RecordFilter type in types.ts defines operator as a strict union of valid operators, but the input validation here uses v.string(). This allows invalid operators to pass validation and silently match (due to the default: return true in matchesFilter). Consider using a union validator to catch invalid operators at the API boundary.

♻️ Suggested fix to validate operators at input level
 filters: v.optional(
   v.array(
     v.object({
       fieldDefId: v.id("fieldDefs"),
-      operator: v.string(),
+      operator: v.union(
+        v.literal("eq"),
+        v.literal("gt"),
+        v.literal("lt"),
+        v.literal("gte"),
+        v.literal("lte"),
+        v.literal("contains"),
+        v.literal("starts_with"),
+        v.literal("is_any_of"),
+        v.literal("is_true"),
+        v.literal("is_false")
+      ),
       value: v.any(),
     })
   )
 ),
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@convex/crm/recordQueries.ts` around lines 270 - 275, The operator field in
the input validator currently uses v.string(), which is too permissive compared
to the RecordFilter operator union in types.ts and lets invalid operators pass
(leading to silent matches via matchesFilter). Replace operator: v.string()
inside the v.object validator with a union/literal validator that enumerates the
exact operator values from the RecordFilter union (e.g., v.union or repeating
v.literal(...) entries matching the operators defined in types.ts) so the API
rejects invalid operators at validation time and stays consistent with
RecordFilter and matchesFilter.

513-525: Consider capping the search limit to prevent excessive reads.

The limit parameter has no upper bound, which could allow callers to request very large result sets. Given the subsequent assembly step queries multiple EAV tables per record, a large limit could exceed Convex's read limits.

♻️ Suggested fix to cap the search limit
-const limit = args.limit ?? 20;
+const MAX_SEARCH_LIMIT = 100;
+const limit = Math.min(args.limit ?? 20, MAX_SEARCH_LIMIT);
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@convex/crm/recordQueries.ts` around lines 513 - 525, The search `limit` is
unbounded (using `args.limit ?? 20`) which can permit very large queries;
enforce a hard cap and validate the input before calling `.take(limit)` on the
`ctx.db.query("records").withSearchIndex("search_label", ...)` chain. Update the
code around the `limit` variable (the `args.limit` usage) to coerce to an
integer, clamp it to a safe maximum (e.g., MAX_LIMIT constant such as 100), and
ensure it is >= 1 (use Math.min/Math.max or equivalent) so the `.take(limit)`
call always receives a sane bounded value.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@convex/crm/recordQueries.ts`:
- Around line 368-370: The cursor parsing using
Number.parseInt(args.paginationOpts.cursor, 10) can yield NaN for malformed
input; update the logic around offset (the variable computed from
args.paginationOpts.cursor in recordQueries.ts) to validate the parsed value and
default to 0 when invalid: parse the cursor, ensure it's a finite non-negative
integer (e.g., check Number.isFinite or !Number.isNaN and value >= 0), and use 0
as the fallback for any invalid or out-of-range values before using offset in
slice operations.

---

Nitpick comments:
In `@convex/crm/recordQueries.ts`:
- Around line 270-275: The operator field in the input validator currently uses
v.string(), which is too permissive compared to the RecordFilter operator union
in types.ts and lets invalid operators pass (leading to silent matches via
matchesFilter). Replace operator: v.string() inside the v.object validator with
a union/literal validator that enumerates the exact operator values from the
RecordFilter union (e.g., v.union or repeating v.literal(...) entries matching
the operators defined in types.ts) so the API rejects invalid operators at
validation time and stays consistent with RecordFilter and matchesFilter.
- Around line 513-525: The search `limit` is unbounded (using `args.limit ??
20`) which can permit very large queries; enforce a hard cap and validate the
input before calling `.take(limit)` on the
`ctx.db.query("records").withSearchIndex("search_label", ...)` chain. Update the
code around the `limit` variable (the `args.limit` usage) to coerce to an
integer, clamp it to a safe maximum (e.g., MAX_LIMIT constant such as 100), and
ensure it is >= 1 (use Math.min/Math.max or equivalent) so the `.take(limit)`
call always receives a sane bounded value.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 1b98835a-77d5-4b5c-a35e-3e24834f107b

📥 Commits

Reviewing files that changed from the base of the PR and between 66ac4c5 and bcf99d1.

📒 Files selected for processing (5)
  • convex/crm/recordQueries.ts
  • convex/crm/types.ts
  • convex/schema.ts
  • specs/ENG-251/chunks/manifest.md
  • specs/ENG-251/tasks.md

Comment thread convex/crm/recordQueries.ts Outdated
- Return null cursor instead of 'done' string when pagination complete
- Add objectDef validation to getRecord (active/system/org checks)
- Add isSystem guard to searchRecords
- Validate and clamp limit parameter (1-100 range)
- Fail-closed on unknown fieldDefId in filters
- Validate pagination cursor format (reject NaN)
- Derive FILTERED_QUERY_CAP from Convex read limits
- Normalize cursor format with tagged prefixes (native:/offset:) across both paths
- Use v.union literals for filter operator validation
- matchesFilter default returns false (fail-closed)
- Handle multi-select fields in is_any_of operator
- Update QueryRecordsResult type to allow null continueCursor

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@Connorbelez Connorbelez merged commit be6c98d into main Mar 30, 2026
1 of 3 checks passed
Connorbelez added a commit that referenced this pull request Apr 20, 2026
### TL;DR

Implements comprehensive record querying and search functionality for the CRM system with EAV (Entity-Attribute-Value) field assembly, filtering, sorting, and full-text search capabilities.

### What changed?

Added three new query functions to handle CRM record operations:

- **`queryRecords`**: Retrieves records with optional filtering and sorting. Uses dual code paths - native Convex pagination for simple queries, and in-memory filtering/sorting for complex queries with a 500-record cap
- **`getRecord`**: Fetches a single record with all field values assembled from EAV tables, plus inbound/outbound linked records via the `recordLinks` table
- **`searchRecords`**: Performs full-text search on record labels using Convex's search index

The implementation includes:
- EAV field assembly that queries only relevant value tables based on field types
- Comprehensive filtering with operators like `eq`, `gt`, `contains`, `is_any_of`, etc.
- Sorting by field values with ascending/descending direction
- Parallel field value loading for performance optimization
- Type definitions for unified record shapes, filters, and linked records
- Search index on the `records` table's `labelValue` field

### How to test?

1. Run `bun check` and `bun typecheck` to verify type safety
2. Execute `bunx convex codegen` to generate updated types
3. Test the three query endpoints:
   - Call `queryRecords` with various filter/sort combinations
   - Use `getRecord` to fetch individual records with their links
   - Try `searchRecords` with different search terms

### Why make this change?

This implements the core querying infrastructure needed for the CRM system to retrieve and display record data. The EAV model requires specialized assembly logic to reconstruct records from normalized value tables, while the dual pagination approach ensures both performance (for simple queries) and functionality (for complex filtered queries). The search capability enables users to quickly find records by their display labels.

<!-- This is an auto-generated comment: release notes by coderabbit.ai -->

## Summary by CodeRabbit

* **New Features**
  * Added record querying with support for field filtering, sorting, and pagination
  * Added record search functionality over label values
  * Added ability to retrieve individual record details with linked entity information
  * Implemented support for outbound and inbound record relationships

* **Chores**
  * Added documentation for record query implementation tasks

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants