ICGC DCC - Configuration Management
Found at: https://github.com/icgc-dcc/dcc-cm
This GitHub repo contains Ansible scripts and roles, as well definitions for Docker containers, all for the purpose of provisioning infrastructure used by DCC.
Even if Ansible and Docker are not something you have a big interest in, they still provide insight into what is required to bring up different components of our infrastructure.
While our Ansible resources contain a multitude of roles, tasks, and playbooks, special attention was given recently to two major portions: The Portal, and the Download Server.
The portal playbook at the moment, will provisions hosts on openstack for nginx, varnish, the data portal itself, elasticsearch, and postgres. The only missing component is actually loading in your data into an elasticsearch index.
The download playbook, provisions also on openstack a small hadoop cluster for the purpose of serving HDFS (One NameNode and 2 DataNodes by default). A host is also provisioned for the download server, and it is configured with the HDFS URI of the new hadoop cluster.
ICGC DCC - Portal Demo
The ICGC DCC Portal Demo is a docker container that allows a user to configure and run a standalone version of the ICGC DCC Portal. The container has all the dependencies requires to run it: DCC Download Server, Elasticsearch and MongoDB. It also provides functionality to download and import ICGC production data.