Pipeline Search Searches EVERY Record in Table Unless Created Date Filter

Curious if there's a better way to do this. Here's an example. I have a table with tens of thousands of records, and each record has a "Date of Work" field that is pulled from a related record. If I want to search for records with that specific "Date of Work", it searches the entire table. Like it pulls in every record and then tries to filter instead of doing a search based on the filter.

So, I search for a date where there should be 400 records, I get a message that it found 400 records hundreds of times. If I change it to also search for a Created Date after a certain date to limit the search even more, it only searches the batch after the created date. So in this case I get that message 40 times instead of 100s of times. This seems incredibly inefficient and makes the pipeline take way longer than it needs to. 

Is there any way to force the search to run against the table and just pull in the results, not a huge batch and then filter from there? I suppose I could manually set up a GET request that uses the API, but why does the built-in search work so inefficiently? I suppose it puts more of the burden on the Pipeline than the app, but that's insane when you're looking for a handful of records in a table with tens of thousands of items. 

Here's an example of what I'm seeing. Just a bunch of "Found 406 Records of Record type" along with the filter.