bundesdata_markup_nlp_data

This is just a repository providing the link to the data used and created by the software from this repository: https://gitlab.ub.uni-bielefeld.de/sporada/bundesdata_markup_nlp_software

Pelase read the description of that project to understand what kind of data this is.

The data can be downloaded here: https://uni-bielefeld.sciebo.de/s/9p55VIn9OLmNqa9

Size: around 70GB

Structure: Note that there are currently two versions of all the data available. Version 1.0 data contains the data described and used for the original master thesis. The master thesis can be viewed here. Version 1.1 data contains new calculated ngrams based on new officlally released xml protocols. Also some fixes have been introduced.

.
├── inputs   ### Data used as input for the software from https://gitlab.ub.uni-bielefeld.de/sporada/bundesdata_markup_nlp_software
│   ├── backup_raw_xml   ### Zip files of all original protocols.
│   ├── current_official_protocols_xml   ### Example file of the new official markup.
│   ├── development_data_xml   ### Set of original xml protocols used for development.
│   ├── faulty_raw_xml   ### All original protocols with errors. The Bundesregierung should have fixed those by now. The Software mentioned above used these faulty ones though because the new ones were not available back then.
│   │   ├── 15_Wahlperiode_2002-2005
│   │   ├── 16_Wahlperiode_2005-2009
│   │   └── 17_Wahlperiode_2009-2013
│   ├── protocols_raw_xml   ### Unziped original protocols.
│   │   ├── 01_Wahlperiode_1949-1953
│   │   ├── 02_Wahlperiode_1953-1957
│   │   ├── 03_Wahlperiode_1957-1961
│   │   ├── 04_Wahlperiode_1961-1965
│   │   ├── 05_Wahlperiode_1965-1969
│   │   ├── 06_Wahlperiode_1969-1972
│   │   ├── 07_Wahlperiode_1972-1976
│   │   ├── 08_Wahlperiode_1976-1980
│   │   ├── 09_Wahlperiode_1980-1983
│   │   ├── 10_Wahlperiode_1983-1987
│   │   ├── 11_Wahlperiode_1987-1990
│   │   ├── 12_Wahlperiode_1990-1994
│   │   ├── 13_Wahlperiode_1994-1998
│   │   ├── 14_Wahlperiode_1998-2002
│   │   └── 18_Wahlperiode_2013-2017
│   └── test_data_xml   ### Set of original protocols fpr testin purposes.
├── MdB_data   ### The official Stammdaten of every MdB can be found here.
├── outputs   ### These are the files an data produced using the software from https://gitlab.ub.uni-bielefeld.de/sporada/bundesdata_markup_nlp_software
│   ├── markup   ### Contains all automatically marked protocols.
│   │   ├── dev_data   ### Automatically marked dev_data protocols.
│   │   │   ├── beautiful_xml   ### Final output: humanreadable automatically marked protocols.
│   │   │   ├── clear_speech_markup   ### Tmp data 
│   │   │   ├── complex_markup   ### Tmp data
│   │   │   ├── new_metadata   ### Tmp data
│   │   │   └── simple_xml   ### Tmp data
│   │   ├── full_periods     ### Automatically marked protocols form all periods.
│   │   │   ├── 01_Wahlperiode_1949-1953
│   │   │   ├── 02_Wahlperiode_1953-1957
│   │   │   ├── 03_Wahlperiode_1957-1961
│   │   │   ├── 04_Wahlperiode_1961-1965
│   │   │   ├── 05_Wahlperiode_1965-1969
│   │   │   ├── 06_Wahlperiode_1969-1972
│   │   │   ├── 07_Wahlperiode_1972-1976
│   │   │   ├── 08_Wahlperiode_1976-1980
│   │   │   ├── 09_Wahlperiode_1980-1983
│   │   │   ├── 10_Wahlperiode_1983-1987
│   │   │   ├── 11_Wahlperiode_1987-1990
│   │   │   ├── 12_Wahlperiode_1990-1994
│   │   │   ├── 13_Wahlperiode_1994-1998
│   │   │   ├── 14_Wahlperiode_1998-2002
│   │   │   ├── 15_Wahlperiode_2002-2005_faulty
│   │   │   ├── 16_Wahlperiode_2005-2009_faulty
│   │   │   ├── 17_Wahlperiode_2009-2013_faulty
│   │   │   └── 18_Wahlperiode_2013-2017
│   │   └── test_data     ### Automatically marked test_data protocols.
│   │       ├── beautiful_xml  ### Final output: humanreadable automatically marked protocols.
│   │       ├── clear_speech_markup   ### Tmp data
│   │       ├── complex_markup   ### Tmp data
│   │       ├── new_metadata   ### Tmp data
│   │       └── simple_xml   ### Tmp data
│   └── nlp   ### All data created from the automatically marked protocols.
│       └── full_periods   ### Contains data created from all protocols.
│           ├── n-grams   ### N-Gramm data based on protocols (sibling of this folder).
│           │   ├── lm_ns_speaker   ### N-grams from lemmatized protocols without stop words counted by speaker.
│           │   │   ├── 1_grams
│           │   │   ├── 2_grams
│           │   │   ├── 3_grams
│           │   │   ├── 4_grams
│           │   │   └── 5_grams
│           │   ├── lm_ns_year   ### N-grams from lemmatized protocols without stop words counted by year.
│           │   │   ├── 1_grams
│           │   │   ├── 2_grams
│           │   │   ├── 3_grams
│           │   │   ├── 4_grams
│           │   │   └── 5_grams
│           │   ├── tk_ws_speaker_(1-3)   ### N-grams from tokenized protocols with stop words counted by speaker.
│           │   │   ├── 1_grams
│           │   │   ├── 2_grams
│           │   │   └── 3_grams
│           │   └── tk_ws_year_(1-4)   ### N-grams from tokenized protocols with stop words counted by year.
│           │       ├── 1_grams
│           │       ├── 2_grams
│           │       ├── 3_grams
│           │       └── 4_grams
│           └── protocols   ### Lemmatized and tokenized protocols used for n-gramm caalculation.
│               ├── protocols_lemmatized_without_stopwords
│               └── protocols_tokenized_with_stopwords
└── protocol_DTD