Hi there, I've been running into this issue and wanted to document it along with some strategies to deal with it.
Env
Observed on dev/pawtucket2x on a collection with over 200,000 objects. Linux/apache2/MySQL.
Steps to repro
- Search for a specific search string like "test"
- Click one of the facets to narrow down your search. I'll select "Audio" under "Object Type" for instance.
- So far so good. I saw a bunch of results with "test" and they narrowed down to just 3 objects with "Audio"
- Clear the files in app/tmp
- Refresh your search.
- Now you'll see that your original search criterion was gone, and only your facet remains, so the result is that facet across the entire collection. In my case, I get all of the nearly 12000 "Audio" objects.
Expected results
I'd expect to see my same search criteria and results if it that could be preserved under this /Search/objects/key/ approach.
If the search can't be preserved, then the resource can't be found. There's a convention on the web for this already: an HTTP 404 status code with a message that the resource can't be found (ideally a customizable message).
Performance impact
Abusive crawlers trawl faceted searches endlessly, even long after the tmp files driving them have been expunged. I can block many of them with a variety of strategies, but the ones that get through are still impacting performance.
If pawtucket could deal with a cache miss with a computationally inexpensive response instead of filtering the whole collection, that would dramatically reduce the performance impact. The bots would land on that 404 in milliseconds instead of minutes, and then they wouldn't have any more facets to "click" on.
Other approaches
Ideally these cache keys wouldn't be displayed in the URLs at all, and the caching mechanism would use the search criteria to generate a key that identifies the search behind the scenes. The URL would show the search criteria, and it would be shareable and durable. But that sounds like it would be a much more difficult change to implement than short-circuiting a cache miss with a 404.
Hi there, I've been running into this issue and wanted to document it along with some strategies to deal with it.
Env
Observed on dev/pawtucket2x on a collection with over 200,000 objects. Linux/apache2/MySQL.
Steps to repro
Expected results
I'd expect to see my same search criteria and results if it that could be preserved under this /Search/objects/key/ approach.
If the search can't be preserved, then the resource can't be found. There's a convention on the web for this already: an HTTP 404 status code with a message that the resource can't be found (ideally a customizable message).
Performance impact
Abusive crawlers trawl faceted searches endlessly, even long after the tmp files driving them have been expunged. I can block many of them with a variety of strategies, but the ones that get through are still impacting performance.
If pawtucket could deal with a cache miss with a computationally inexpensive response instead of filtering the whole collection, that would dramatically reduce the performance impact. The bots would land on that 404 in milliseconds instead of minutes, and then they wouldn't have any more facets to "click" on.
Other approaches
Ideally these cache keys wouldn't be displayed in the URLs at all, and the caching mechanism would use the search criteria to generate a key that identifies the search behind the scenes. The URL would show the search criteria, and it would be shareable and durable. But that sounds like it would be a much more difficult change to implement than short-circuiting a cache miss with a 404.