Detecting if a certain text is JavaScript

user_11833025 0
Viewed 104

I have a text file. Its content may be plain text or JavaScript source code. I need an efficient and high accuracy approach to detect if the text file contains plain text or JavaScript code.

My approach: I tried to extract syntax-specific keywords, punctuations, operators, regex based loop detection for this. I added all the values and divided it by the length of the file and got a specific ratio. Let's say the ratio is greater than 0.2 then the file is JavaScript else plain text. The problem I ran in was with regex based loop detection. It was taking too much time for file size > 1 Mega Bytes, hence I abandoned the approach.

Any resources or approaches would be highly appreciated. Thank you.

nlp python javascript
1 Answers

You could use NLP in a slightly different manner — read the entire js file, tokenize it and build a frequency table of tokens; use this data to apply the logic you have outlined. I have tried to do this using wink-nlp on a 6.6MB JS file and it produced the frequency table in about 5 seconds on an intel 2.2GHz i7 macbook pro. Here is the Node.js code for your reference:

const fs = require( 'fs' );
const winkNLP = require( 'wink-nlp' );
const model = require( 'wink-eng-lite-web-model' );
const its = require( 'wink-nlp/src/its.js' );
const as = require( 'wink-nlp/src/as.js' );

const rawJS = fs.readFileSync( '<your-js-file-with-path>', 'utf8' );

const nlp = winkNLP( model );

const doc = nlp.readDoc( rawJS );
// The const ft has the frequency table:
const ft = doc.tokens().out( its.normal, as.freqTable );

Here is the partial output:

[ '\n', 160960 ],
[ '.', 125520 ],
[ '(', 108440 ],
[ '/', 106480 ],
[ '=', 95040 ],
[ ')', 74800 ],
[ ';', 64120 ],
.
.
[ '{', 16640 ],
[ '}', 16640 ],
[ 'const', 15920 ],
.
.
user_110687980
Related