Go to file
2020-05-19 10:25:16 +02:00
app Add legal notice and gitlab link as button 2020-05-19 10:25:16 +02:00
migrations First working mail notification system 2020-05-14 15:30:13 +02:00
tests Make inner package imports relative 2020-03-27 12:06:11 +01:00
.gitignore Add mac os files to .gitignore 2020-03-26 17:31:39 +01:00
.gitlab-ci.yml Update 2019-11-04 15:06:54 +01:00
config.py Change some email settings. MAIL_DEFAULT_SENDER removed. 2020-04-20 14:24:36 +02:00
docker-compose.yml Let daemon depend on web container, to make sure that db migrations are applied before daemon starts. 2020-05-19 09:27:21 +02:00
docker-entrypoint.sh remove gunicorn from entrypoint 2020-04-27 10:20:47 +02:00
Dockerfile Fix Ports 2020-04-20 10:40:10 +02:00
nopaque.env.tpl Merge notify into daemon 2020-05-15 14:43:12 +02:00
nopaque.py Add all necessary models to flask shell context 2020-05-18 14:47:04 +02:00
README.md Add some Readme stuff 2020-03-16 10:49:45 +01:00
requirements.txt Switch back to socketio server while development 2020-04-27 09:34:12 +02:00

Opaque

Opaque is a virtual research environment (VRE) bundling OCR, NLP and additional computer linguistic methods for research purposes in the field of Digital Humanities.

Opaque is designed as a web application which can be easily used by researchers to aid them during their research process.

In particular researchers can use Opaque to start OCR jobs for digitized sources. The text output of these OCR jobs can then be used as an input for tagging processes (POS, NER etc.).

As a last step texts can be loaded into an information retrieval system to query for specific words, phrases in connection with linguistic features.

Dependencies

  • cifs-utils
  • Docker
  • Docker Compose

Configuration and startup

  1. Create Docker swarm:

The following part is for users and not the development team. The development team uses a script which sets up a local development swarm.

The generated computational workload is handled by a Docker swarm. A swarm is a group of machines that are running Docker and joined into a cluster. It consists out of two different kinds of members, managers and workers. Currently it is not possible to specify a dedicated Docker host, instead Opaque expects the executing system to be a swarm manager of a cluster with at least one dedicated worker machine. The swarm setup process is described best in the Docker documentation.

The dev team can use dind_swarm_setup.sh. If the workers cannot join the manager node. Try opening the following ports using the ubuntu firewall ufw:

sudo ufw allow 2376/tcp \
&& sudo ufw allow 7946/udp \
&& sudo ufw allow 7946/tcp \
&& sudo ufw allow 80/tcp \
&& sudo ufw allow 2377/tcp \
&& sudo ufw allow 4789/udp

sudo ufw reload && sudo ufw enable
sudo systemctl restart docker
  1. Create a network storage: The dind_swarm_setup.sh script handles this step for the dev team aswell.

A shared network space is necessary so that all swarm members have access to all the data. To achieve this a Samba can be used.

# Example: Create a Samba share via Docker
# More details can be found under https://hub.docker.com/r/dperson/samba/
sudo mkdir -p /srv/nopaque/storage
docker run \
    --name opaque_storage \
    -v /srv/nopaque/storage:/srv/nopaque/storage \
    -p 445:445 \
    dperson/samba \
      -p \
      -s storage.nopaque;/srv/nopaque/storage;no;no;no;nopaque \
      -u nopaque;nopaque

# Mount the Samba share on all swarm member nodes with the following code
sudo mkdir /mnt/nopaque
sudo mount --types cifs --options gid=${USER},password=nopaque,uid=${USER},user=nopaque,vers=3.0 //<YOUR IP>/storage.nopaque /mnt/nopaque
  1. Download Opaque
git clone https://gitlab.ub.uni-bielefeld.de/sfb1288inf/opaque.git
cd opaque
docker-compose pull
  1. Configure your instance: For production environments it is recommended to activate and secure the Docker HTTP API. You can read more here.
mkdir logs
cp nopaque.env.tpl nopaque.env
<YOUR EDITOR> nopaque.env # Fill out the empty variables within this file. For the gitlab login either use your credentials (not recommended) Or create a gitlab token
  1. Further development instructions Use the following command to allow docker to pull images from your gitlab registry. TODO: Check if this could also work wit a token?
docker login gitlab.ub.uni-bielefeld.de:4567
  1. Start your instance
# Execute the following 3 steps only on first startup
docker-compose run web flask db upgrade
docker-compose run web flask insert-initial-database-entries
docker-compose down

docker-compose up
  1. Alter Database Models
docker-compose run web flask db migrate
docker-compose run web flask db upgrade