Running a Software Project#
At ICHEC you may be responsible for running a project that involves developing software.
Some important considerations are:
where will you store this software as it is being developed?
how will people in your group and others more widely collaborate on the software?
how will the software be delivered to its end-users?
how do you make sure changes you make won’t break functionality for users?
how do you avoid security or licensing issues?
This guide has some recommendations on how to manage the software you are developing, covering each of the questions above.
Where will you store this software as it is being developed?#
All software developed at ICHEC SHOULD be:
in version control
stored on our ICHEC Gitlab, unless there is a reason not to (covered below)
Version control (e.g. git) makes sure that your valuable effort developing the software is not accidently lost or overriden. It is foundational to other tools that you will need to deploy, share and secure your software.
Storing code on the ICHEC Gitlab means that it won’t be lost and that it is easy for colleagues to find it, learn from it and collaborate with you. It also means that you can easily benefit from pre-existing tools and infrastructure built by other groups in ICHEC.
The are reasons why you may not be able to use the ICHEC Gitlab:
you are contributing to software that is being managed elsewhere and not maintained by ICHEC
your project is part of a consortium that mandates where its software is stored
you anticipate that several external collaborators will want to contribute to the software, including code or defect reports
For 2) and 3) our Gitlab supports automatic mirroring which may resolve the issue in some cases.
Regardless of the exceptions above, code SHOULD NOT be stored locally on a machine (including production servers), in Teams, OneDrive or Sharepoint.
How will people collaborate on the software?#
Collaborators on a piece of software can be:
within your own group or project
in ICHEC
outside of ICHEC
Collaboration can take the form of:
writing code and submitting merge requests or patches
reviewing submitted code
development of the project work plan
reporting defects
requestig features
Project Visibility and Membership#
You SHOULD review and understand Gitlab’s ‘project visibility’ and ‘membership’ features for your project.
These featyres allow control over WHO can collaborate on a project and WHAT form of collaboration they can be involved in.
If you lock down your project too heavily on Gitlab, i.e. making it Private, this will limit the amount of internal collaboration - encouraging siloing.
If you make it too open or public you may expose sensitive IP or project configuration, affecting security. If you are unsure about IP or security concerns you can make your project ‘Internal’, so that only ICHEC users can see it. If you are comfortable regarding these issues you can make your project ‘Public’. That can make life easier when working with external collaborators and open-source infrastructures.
Issue Tracker#
You SHOULD use the Gitlab issue tracker to track ongoing and planned work on the software. You SHOULD use merge requests to make changes to a codebase.
You can also use the issue tracker for more general planning, including prioritization and milestones. You may require a more feature-rich project management tool for large or more complex projects however.
How will the software be delivered to its end-users?#
You should think about how your end-users or stakeholders will use your software early in the project - otherwise you may build a solution that is difficult or impossible to deploy, close to a deployment deadline.
Below are some typical ways software may be delivered.
Url to a repository#
In this case you will just provide your end-users a hyperlink to the code repository.
This may be relevant if it is useful for them to inspect or build on the code itself. If you have a Python package definition or Docker container spec in a standard location they may also be able to build a package or container just from the repo.
In this case you should take care to provide some form of versioning and release management via, for example, git tags so that they can build on a particular version of your software.
Application package or binary#
You may need to provide you application as a binary or package that will execute on someone’s system. The format will depend on your adopted software language. For example, if it is Python with relatively simple dependencies you can provide a Python package. If it is compiled code you may wan to provide packages for particular operating systems or Linux distros, such as an RPM or DEB file.
When packaging Python or similar, you need to be careful regarding dependencies. It is easy to make a Python package that is missing runtime dependencies or has some that are difficult to install. This will make using your software difficult, or even impossible, for your end users. Often installing Python dependencies on Windows is painful for users, you may prefer to use a container if unsure.
When passing application binaries to users you SHOULD NOT build them on your local machine - this opens your users to security issues. They should be built on a secure CI pipeline.
Containerized application#
If your application will be deployed in a web setting, has non-trivial dependencies or needs to be reproducible you should consider passing along a container. This can involve producing a Containerfile (Dockerfile), may involve building a container image and storing that image in an online repository.
You SHOULD NOT build container images for use by others on your local machine- this exposes users to security issues. They should be built in a secure CI pieline.
Web service#
Your software may be delivered to users as a web service - which may take the form of a UI, portal or dashboard and/or an API they can interact with.
In this case you should containerize your application and make sure admin credentials are stored separately, for example in a ‘env’ file at runtime.
The container SHOULD be deployed behind a managed reverse proxy server and itself run the application on a production-ready server, such as gunicorn with Python. You SHOULD NOT run Flask, Django or similar directly in the container. Their servers are not production-ready nor secure.
How do you make sure changes won’t break functionality for users?#
All software projects SHOULD have automated tests. Even research-oriented ones.
Automated tests ensure the base functionality of your software remains functional while you make changes.
Without tests it is very easy to break functionality without noticing it - especially approaching deadlines when there is time pressure and perhaps extra collaborators brought onto a project.
How do you avoid security or licensing issues?#
You SHOULD have an approriate license accompanying your software, so others can use it.
If you are including external software you MUST understand and respect its licensing terms.
Here is a sample, non-exhaustive security checklist:
there are no hard-coded credentials or sensitive config in the project repos
gitignore and dockerignore files are set so sensitive files aren’t included in repos or containers
web servers are running in production rather than debug mode
views are suitably protected with login decorators
up-to-date software versions and dependencies are being used
throttling and IP blocklisting are in place
databases are not exposed to the internet
user input is validated
directory traversal is disabled on the application server
GitLab Development Platform#
ICHEC uses GitLab as a development platform for projects. Although originally associated with version control (git) this platform (which is similar to GitHub) has a variety of tools available for delivering software projects, including:
Version control via git
Agile project management (Scrum or Kanban)
Continuous Integration and build systems
Release, Deployment and artifact management
Secret and credential management
Documentation publishing and wiki
It is a powerful resource - it is well worth understanding everything it can do, it is easy to unnecessarily build something already available on this platform. For more details see the development platforms section
Finding Projects#
You will likely be assigned some projects to work on and will be given direct URLs to their GitLab page. You can also explore projects and search by group and activity.
It is good to use the explorer to research on-going and previous ICHEC projects to benefit from other’s experience and share knowledge across the centre.
Contributing to Projects#
Each project should have a README at its top level - which you can see when navigating to it in GitLab. It may have specific details on how to contribute and who to contact if interested.
You can also refer to the version control guide to understand working with git and GitLab overall.
Ultimately, a typical contribution flow would involve:
Getting assigned or creating a ‘GitLab Issue’ which describes a planned change
Checking out a local copy of the repository
Creating a ‘working branch’ using git
Making your change (e.g. modifying some code)
Push your ‘working branch’ back to GitLab
Creating a ‘Merge Request’ to merge your branch into a development branch, referencing the ‘GitLab Issue’.
Merging your code and close the ‘GitLab Issue’
Starting and Managing a Project#
When starting a new project or managing one it may be helpful to refer to the starting a project with GitLab page for some tips and best-practice.
Local Development Tooling#
Most ICHEC staff will be working on Mac, either an Intel (older) or Arm (newer) version. You can check the getting started on Mac guide for setting up a development environment.
Deploying Projects#
Depending on your project needs you may ‘deploy’ simply by uploading a binary somewhere, or you may need to push to a live distributed system.
The ‘deployment guide’ gives some information on different deployment approaches.