Repository navigation
fix: do not look up TfIdf terms on the prototype of documents and cache - #785
Open
OsamaAnsar wants to merge 1 commit into
Open
OsamaAnsar wants to merge 1 commit into
OsamaAnsar wants to merge 1 commit into
Conversation
Terms that match a property of Object, such as "constructor", were looked up on the prototype chain. TfIdf.tf returned the Object function for a document that lacks the term, so tfidf() returned NaN, and the IDF cache created by the constructor and addFileSync was a plain object, so idf() returned the Object function for instances created from deserialized data or filled with addFileSync. Use own property lookups for the term frequency and use a cache without a prototype in the constructor and in addFileSync, like addDocument does.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes # (no existing issue; the problem is described below)
What was wrong
Terms that are also property names of
Object, such asconstructor, were looked up on the prototype chain inlib/natural/tfidf/tfidf.js:TfIdf.tfisdocument[term] ? document[term] : 0. On a plain-object document that lacks the term,document['constructor']is theObjectfunction, sotfreturns a function andtfidf()returnsNaN.addFileSyncsetthis._idfCache = {}. With a plain-object cache,idf('constructor')findsObjectin the cache and returns the function instead of a number. This hits instances created from deserialized data (new TfIdf(json)) and instances afteraddFileSync()(addDocumentandremoveDocumentalready useObject.create(null); issue Tfidf scoring doesnt work for the word 'constructor' #119 only patched the document-building side).Real output on current
master(3 documents:this document is about node and javascript,this document is about ruby and rails,the constructor is a special method):tfidf('constructor', 0)(term absent)NaN0tfidfs('constructor')[NaN, NaN, 1.4054651081081644][0, 0, 1.4054651081081644]tfidf('__proto__', 0)NaN0typeof idf('constructor')functionnumbertfidf('constructor', 0)(absent)NaN0tfidf('constructor', 2)(present)NaN1.4054651081081644addFileSync:typeof idf('constructor')functionnumberaddFileSync:tfidf('constructor', 1)(present)NaN1Ordinary terms (e.g.
node) give identical results before and after. (tfidfsreturnsNaN;JSON.stringifyof it printsnull.)Proposed Changes
TfIdf.tfanddocumentHasTermonly consider own properties of the document (Object.prototype.hasOwnProperty.call).addFileSynccreate the IDF cache withObject.create(null), asaddDocumentandremoveDocumentalready do.spec/tfidf_spec.tsfor:tfof an absent reserved name,tfidfof 0 (notNaN) for absent reserved names (toString,valueOf,hasOwnProperty,constructor,__proto__, ...), a presentconstructorterm,listTermswith noNaN, a deserialized instance and the cache afteraddFileSync().Verification
lib/natural/tfidf/tfidf.jsreverted (confirmed withgit diff --statand grep), 4 of the new specs fail (thetf,tfidfof absent terms, deserialized andaddFileSyncones); the other new specs are regression guards that pass either way. With the change restored, all 27 specs intfidf_specpass.npm test: 864 specs, 0 failures.eslint . --ext .ts(thelintscript): no errors.npx standard lib/natural/tfidf/tfidf.js: no errors.npm run test_iowas not run (it needs external services).Checklist before merging
index.d.tsfiles (not applicable: no API or type change,index.d.tsis untouched)