Bitbucket DockerOps

Help and business engagements

For any requirements you can contact me at [email protected] for business engagements.

Bitbucket Server

This docker-compose project contains the following services:

Bitbucket server
postgres db
backup and restore scripts based on a "run once" docker container

Bitbucket server is based on official Atlassian parent image.

For semi-automated rollout, Ansible is used.

Required local setup

Your mileage may vary since there are different distros and operating systems. The general idea is to have:

Python
Ansible
Git
An editor, such as VS Code with Extensions such as WSL, Ansible, Python, Pylance, YAML
pyenv to allow you to run other versions of python and ansible
ansible-lint and other packages using the pyenv command

Notes: For python and ansible, in order to run the playbook, it's important to have a compatible version of python and ansible with the version on the server. Sometimes the newer versions of python and ansible won't allow you to run playbooks on the server and will spit out errors.

Cygwin

Ansible has to be installed locally in order to run the playbook. One way to do so on Windows is to use it via cygwin. Install the "ansible" package in your cygwin installation.

WSL Linux

Another way is to request WSL Linux in your Windows environment. This will allow you to install Ubuntu or any other linux environment with a linux/bash terminal you might feel more comfortable.

General recommendations with WSL Linux and Ansible

With WSL Linux I would recommend installing:

VS Code: https://code.visualstudio.com/
In WSL Linux terminal install pipx, python3, and git (your mileage may vary if you choose different distro othen than Debian/Ubuntu): sudo apt install git git-lfs git-all python3 pipx -- Source: https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html#pipx-install
pipx install --include-deps ansible ansible-lint
Add in your ~/.bashrc:

export ANSIBLE_CONFIG=./ansible.cfg
export PATH="$HOME/.local/bin:$PATH"

You can then install ansible, python, docker and other extensions in VS Code to enable code checks / lint etc.

Before running the playbook - prepare the environment

Working with ansible and older python (3.6.8) has its downsides. You should install compatible versios for ansible, ansible-base, ansible-lint to make everything work as expected. This goes beyong the scope of this README file, but you must prepare your environment to match the environment of the server (i.e. same python version). Otherwise, expect issues while running the ansible playbook.

Prepare WSL Linux for ansible playbook

Install the following in your WSL Linux:

sudo apt-get install -y make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev xz-utils tk-dev libffi-dev liblzma-dev python-openssl git
sudo apt install clang -y
curl https://pyenv.run | bash
pyenv update
pip install --upgrade pip
pyenv rehash
CC=clang pyenv install 3.6.8

Prepare your ~/.bashrc

Prepare your ~/.bashrc file:

export ANSIBLE_CONFIG=./ansible.cfg
export PATH="$HOME/.local/bin:$PATH"
export PATH="$HOME/.pyenv/bin:$PATH"
export PYENV_PYTHON_BUILD_CACHE_PATH="$HOME/.pyenv_cache"
export PYENV_ROOT="$HOME/.pyenv"
#[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(~/.pyenv/bin/pyenv init -)"

# Fix Ansible
export ANSIBLE_PYTHON_INTERPRETER="$(which python)"

Configure pyenv to install compatible python version

Configure your local system to install compatible python version. In our case we needed to match python version 3.6.8, so we found the compatible package versions of ansible and dependencies that match that version of python:

pyenv virtualenv 3.6.8 py368
pyenv virtualenvs
pyenv rehash
pyenv activate py368

CC=clang pip install --upgrade pip
pyenv rehash
CC=clang pip install wheel
pyenv rehash
CC=clang pip install cryptography==3.4.8
pyenv rehash
CC=clang pip install ansible-base==2.10.1
pyenv rehash
CC=clang pip install ansible==2.10.0
pyenv rehash
CC=clang pip install ansible-lint==4.3.7

That should be it.

Test using:

python --version
ansible --version

Secret files

Secret files cannot be part of the image, as the docker image gets pushed to the docker registry. Moreover, different files might be needed across the stages (DEV, QA, PRD). Thus, all secret files get mounted as a separate volume from the secrets folder. The same is valid for SSL certificates and ssl keystores.

NOTE about encrypting files - ansible vault

Each file in the secrets/ and certs/ subfolder needs to be encrypted with ansible-vault prior to committing to git.

Ansible vault also requires a password to encrypt/decrypt prior deploying. You can request it from the current mentor/developer in charge of the Bitbucket app.

Protect against unintended commits

To avoid unintended commits of these files, you have to configure your git to look for hooks in the hooks folder of a project.

Execute locally the following command in the root of the project:

git config core.hooksPath hooks

so that a pre-commit hook defined there checks all your files located in any secrets and certs subfolder. If they are not encrypted, the commit will fail.

Required files for build process

docker-compose level

database_secret # database password
bitbucket_secret # bitbucket password

Dockerfile level

Intermediate.crt Intermediate certificate to be added to the cacerts keystore
Root_CA.crt Root certificate CA to be added to the cacerts keystore

Backup and restore

Things to request by GSD ticket

copy following systemd files to /etc/systemd/system
- bitbucket-backup.service
- bitbucket-backup.timer
- execute 'systemctl daemon-reload'
- execute 'systemctl enable bitbucket-backup.timer'
- execute 'systemctl start bitbucket-backup.timer'

Scheduled backup

There is a systemd service unit called bitbucket-backup.service created, that just triggers a shell script bitbucket-backup.sh. See docker/bitbucket/bitbucket-backup.sh for more details.

The service unit is being triggered by a systemd timer called bitbucket-backup.timer every night at 4:00:00. To check the status of the backup service, just call sudo systemctl status bitbucket-backup.service, to check the status of the timer call sudo systemctl status bitbucket-backup.timer. To check the recent log output of the service, use sudo journalctl -u bitbucket-backup.

Bitbucket DIY Backup information

Bitbucket DIY Backup (and restore) are documented at: https://confluence.atlassian.com/bitbucketserver/bitbucket-diy-backup-776640056.html

Bitbucket DIY Backup system latest source code is found at: https://bitbucket.org/atlassianlabs/atlassian-bitbucket-diy-backup/src/master/

Main running bash scripts are:

for backup: bitbucket-diy-backup/bitbucket.diy-backup.sh
for restoration: ``bitbucket-diy-backup/bitbucket.diy-restore.sh`

Configuration file for backups: bitbucket-diy-backup/bitbucket.diy-backup.vars.sh

You have examples such as bitbucket-diy-backup/bitbucket.diy-backup.sh.example and bitbucket-diy-backup/bitbucket.diy-backup.vars.sh.example-postgres-rsync

System setup

Backup mount

The network drive to store the backups on should be mounted at /opt/backup

NEW backup setup

Systemd setup

There are following unit files located at /etc/systemd/sytem/

bitbucket-backup.service - the service unit file that runs the script /opt/docker/bitbucket/bitbucket-backup.sh
bitbucket-backup.timer - the systemd timer unit that triggers the bitbucket-backup.service unit
if you require to do any changes in this service, you have to reload the systemd daemon: sudo systemctl daemon-reload (with adequate sudo required permissions of course, or request from unix team or someone with root permissions).

bitbucket-backup.sh and bitbucket-diy-backup-run.sh scripts

This script is a wrapper script that is runs as a systemd service.
The script is located at /opt/docker/bitbucket/
It mounts the backup drive if necessary
Then runs bitbucket-diy-backup-run.sh, another wrapper script, which creates client docker container (runs once and removes itself) using docker run command. Then runs the backup-diy-backup.sh script
The backup is being done in a docker-related /tmp path (not system /tmp path), then compressed from the containers' volume /tmp folder onto the backup drive.
At the end it deletes backups that are older than 30 days.

Bitbucket restore guide

Bitbucket restore ansible playbook

An ansible playbook to restore an earlier backup exists: restore.yaml

Preparation

Make sure you have the correct configuration set. Usually there's nothing to be changed. See bitbucket-diy-backup/bitbucket.diy-backup.vars.sh

Restore bash script

Bash script being ran: bitbucket-diy-restore-run.sh

You can run the script remotely or locally. It will restore the latest available backup file.

The original ./bitbucket.diy-restore.sh in bitbucket-diy-backup/ folder also accepts the name of backup file being set manually, so if you want a manual backup file (choosing your backup file), you can do so by running the bitbucket-diy-backup/bitbucket.diy-restore.sh and see the available backups, then ./bitbucket.diy-restore.sh <backup-file-name-from-list>

HOWTO

Run the playbook

ansible-playbook -i inventory/dev.yaml site.yaml -u <YOURUSERHERE> --ask-pass --vault-password-file .ansible/vault-pw

The parameters -u <YOURUSERHERE> --ask-pass can be omitted when you have created your user with ssh key and passwordless sudo enabled.

If you want to login with a specific ssh key (example path to private key file is ~/.ssh/id_ed25519):

ansible-playbook -i inventory/dev.yaml site.yaml -u <YOURUSERHERE> --vault-password-file .ansible/vault-pw --private-key ~/.ssh/id_ed25519

If you want to check the playbook before actually running it use the --check argument:

ansible-playbook --check -i inventory/dev.yaml site.yaml -u <YOURUSERHERE> --vault-password-file .ansible/vault-pw --private-key ~/.ssh/id_ed25519

Working with Ansible Vault

Encrypt a file

ansible-vault encrypt --vault-password-file .ansible/vault-pw encrypted.yml

Encrypt all files in a folder

for i in $(find ./docker/bitbucket/secrets ./certs -type f); do ansible-vault encrypt --vault-password-file .ansible/vault-pw $i && echo $i; done

Decrypt a file with ansible-vault

ansible-vault decrypt --vault-password-file .ansible/vault-pw encrypted.yml

Note about updating SSL certificates

This folder contains SSL certificates exported as Java Keystore files, that are the only accepted one by the application.

Omitting the Root chain might cause the application links between the servers to fail due to unknown/self-signed certificate error!

Name		Name	Last commit message	Last commit date
Latest commit History 1 Commit
backup		backup
certs		certs
docker/bitbucket		docker/bitbucket
group_vars		group_vars
hooks		hooks
inventory		inventory
.ansible-lint		.ansible-lint
.gitignore		.gitignore
.python-version		.python-version
README.md		README.md
ansible.cfg		ansible.cfg
readme-update-toc.sh		readme-update-toc.sh
restore.yaml		restore.yaml
runme.dev.example.sh		runme.dev.example.sh
runme.prod.example.sh		runme.prod.example.sh
runme.qa.example.sh		runme.qa.example.sh
site.yaml		site.yaml
unlock.yaml		unlock.yaml

medigeek/bitbucket-docker-compose-ansible

Folders and files

Latest commit

History

Repository files navigation