NLP - Natural Language Processing

This software implements a heavily parallelized pipeline for Natural Language Processing of text files. It is used for nopaque's NLP service but you can also use it standalone, for that purpose a convenient wrapper script is provided.

Software used in this pipeline implementation

• Official Debian Docker image (buster-slim) and programs from its free repositories: https://hub.docker.com/_/debian
• pyFlow (1.1.20): https://github.com/Illumina/pyflow/releases/tag/v1.1.20
• spaCy (3.0.5): https://github.com/tesseract-ocr/tesseract/releases/tag/4.1.1
• spaCy medium sized models (3.0.0):
  • https://github.com/explosion/spacy-models/releases/tag/de_core_news_md-3.0.0
  • https://github.com/explosion/spacy-models/releases/tag/en_core_web_md-3.0.0
  • https://github.com/explosion/spacy-models/releases/tag/it_core_news_md-3.0.0
  • https://github.com/explosion/spacy-models/releases/tag/nl_core_news_md-3.0.0
  • https://github.com/explosion/spacy-models/releases/tag/pl_core_news_md-3.0.0
  • https://github.com/explosion/spacy-models/releases/tag/zh_core_web_md-3.0.0

Use this image

1. Create input and output directories for the pipeline.

mkdir -p /<my_data_location>/input /<my_data_location>/output

2. Place your text files inside /<my_data_location>/input. Files should all contain text of the same language.

3. Start the pipeline process. Check the pipeline help (nlp --help) for more details.

# Option one: Use the wrapper script
## Install the wrapper script (only on first run). Get it from https://gitlab.ub.uni-bielefeld.de/sfb1288inf/nlp/-/raw/1.0.0/wrapper/nlp, make it executeable and add it to your ${PATH}
cd /<my_data_location>
nlp -i input -l <language_code> -o output <optional_pipeline_arguments>

# Option two: Classic Docker style
docker run \
    --rm \
    -it \
    -u $(id -u $USER):$(id -g $USER) \
    -v /<my_data_location>/input:/input \
    -v /<my_data_location>/output:/output \
    gitlab.ub.uni-bielefeld.de:4567/sfb1288inf/nlp:1.0.0 \
        -i /input \
        -l <language_code>
        -o /output \
        <optional_pipeline_arguments>

4. Check your results in the /<my_data_location>/output directory.