Actually, this applies to most CI/CD tooling but I am using Team City so let's start here. I assume you know basically what the CI/CD tool is doing and how you have some hierarchy of projects, jobs, tasks and channels.

For my post, I will call a Project something that links to a single code repository. A job can be described as a set of tasks that are run against a project so you might have one job per project or you might have e.g. CI and FullTest builds triggered differently. A task is a single unit of work.

The main end-game for Devops is to automate as much as possible since that reduces your workload for deployment, it finds problems quickly and it allows you to repeat good stuff multiple times with almost no risk and bad things are magnified quickly so they are easier to find.

1) Do your homework for a good tool

Probably most of the CI/CD tools can handle most project types, since almost all of them have the ability to run command-line, powershell, bash etc. to do anything that is not built-in. This is good but is also a reason for the developers to be slower at releasing features. You want .Net support? Write it in a script! Some are much better than others for certain frameworks so look into it, the time you might save could be measured in weeks if you can avoid writing loads of hard-to-maintain powershell scripts just to do something that should be simple.

2) Carefully think about project layout

A common question is whether to have a single giant project or to split things into smaller projects. Smaller projects build quicker but then you have depedency problems and the latency of making a library change, waiting for a build, then doing a change in the consuming project that might or might not work. Sure, you can temporarily import the library into your work area while testing stuff out but that is asking for accidents to happen.

Build (and test) times are key so smaller is generally better. We have a fairly simple app that runs a good level of functional tests (50%ish - could be better) and that takes 20+ minutes to run. Currently, we don't have capacity to build feature branches so that is a long time to realise you broke something. Reducing the risk by separating libraries and running their own unit tests first should help with reducing build times.

3) Have a fast way to deploy new build agents

Many CI/CD tools have a limited number of agents before you start paying the big dollars but actually, paying a few hundred pounds for the extra clout might be worth it when you are up against a deadline but it is no use if you cannot deploy new agents quickly.

Team City, for instance, can deploy an agent using push, which is relatively painless except it can only auto-install a limited number of tools like DotCover. From scratch, installing a new agent onto Windows Server for a .net build (node, nunit, dotnet core etc.) takes a good few hours. The solution? Find a good virtual appliance or build your own and keep it somewhere safe.

We keep a snapshot of a VM on the cloud which can be used to run up new servers relatively quickly (5 minutes ish) and all it requires is occasional updates and a re-snapshot.

4) Make sure your steps work on remote agents

It is common to start by putting everything onto one server and getting it work. We did this and as soon as I added a new remote build agent, a load of the builds wouldn't run. Some of the causes were easy to fix, others having taken more effort. These causes include:
1) Hard-coded paths in the tasks. Use build agent parameters instead.
2) Resource code on the build server that is copied in to the build. This should be part of the project if it is necessary and pulled in from the source checkout.
3) Accessing netbios paths and other internal areas. Use nuget, myget, sftp or built-in tasks to ship code back from the agent. Team City supports automatic artifact uploads but this requires that you setup the artifacts correctly.
4) Tests that access a dev database. We do not want to make our database publically accessible so we need to change the build to deploy a temp database to use for functional testing.
5) Any tests that are locale sensitive. We shouldn't have any but a few were assuming what the default locale was, which then failed on the build agent despite trying to set its locale correctly.

5) Keep your config in source control

Most of us hopefully plan for a web server going down and can redeploy quickly but what happens if our build server goes down? Link your CI/CD tool to source control so you get visibility of changes and can quickly redeploy a new instance. You can also make changes directly in the config files and get the build server to automatically apply them the next time it runs e.g. adding some new dlls to the unit test task.

6) Learn to read the build logs

Most build logs are extremely verbose and are not always easy to understand when an error occurs. This only comes with time but learning this can make your job a lot easier when debugging!