sporada/nlp
1
0
Fork 0
mirror of https://gitlab.ub.uni-bielefeld.de/sfb1288inf/nlp.git synced 2024-12-27 13:24:18 +00:00
Code Issues Projects Releases Wiki Activity

Remove character limit.

Browse Source
This commit is contained in:
Stephan Porada 2019-03-06 12:19:43 +01:00
parent e578f5a8ff
commit ff1e0a51c4
1 changed files with 0 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
spacy_nlp
View File

@ -32,10 +32,6 @@ SPACY_MODELS = {"de": "de_core_news_sm", "en": "en_core_web_sm",
# Set the language model for spacy # Set the language model for spacy
nlp = spacy.load(SPACY_MODELS[args.lang]) nlp = spacy.load(SPACY_MODELS[args.lang])
# Set maximum character length for input documents. According to documentation
# every value above 1000000 (1 Million) can cause memory allocation errors.
# We are testing it with 10 Million for now.
nlp.max_length = 10000000 # 10 Million character limit
# Read text from the input file # Read text from the input file
with open(args.input) as input_file: with open(args.input) as input_file: