GIT: Downloading Data from a Remote

The "fetch" command is indispensable for staying up-to-date in a project: only when performing a "git fetch" will you be informed about the changes your colleagues pushed to the remote server.

git fetch Updates Your View of Remote Data

Before we talk about "git fetch" itself it's really important to understand how remote data is managed in Git.

When looking at your remote branches and commits, you must keep in mind that you're looking at a "snapshot" of data. There is no "live" connection to your remote server.
Instead, Git saves the latest information about your remotes each time a "fetch" operation is performed. This means that the information about remote branches and commits that you see is only as fresh as the last snapshot you fetched.

To see new remote branches, commits, tags, etc., you have to explicitly tell Git to download this information - with "git fetch".

Fetch Does Not Integrate

Fetch is probably one of the most harmless commands in Git: although it downloads data to your local repository, it does not integrate any of that data into your local branches or working copy. Fetch really only updates your view on remote data.

It's up to you to later integrate any of the new data into your local project.

git fetch in Action

The actual syntax of a Fetch command is dead simple: "git fetch" and the name of the remote server is all we need.

$ git fetch origin

A useful parameter, however, is "prune": this tells Git to remove any references to branches that no longer exist on the remote. This ensures that you don't look at stale data:

$ git fetch origin --prune

GIT: How to Integrate Branches

Separating different topics into different branches is a crucial practice for any serious developer. By not mixing up code from one feature / bugfix / experiment with another, you avoid a lot of problems - and don't have to worry about breaking things in your development branches.

But at some point your code will (hopefully) reach a state where you'll want to integrate it with the rest of the project. This is where the "git merge" command comes in.

A Simple Example

Let's assume a very simple example scenario:

Our goal is to integrate the changes from "contact-form" back into "master".

Preparing to Merge

Before merging your code into one of your project's long-running branches (like a "development" or "master" branch), make sure that your local repository is up to date. Both your local feature / bugfix / <topic> branch and the receiving branch should be updated with the latest changes from your remote server.

Start with a "git fetch", followed by a "git branch -va":

$ git fetch
...

$ git branch -va
  master             87eab46 [behind 1] Fix #332
* contact-form       b320ab3 Ensure safe login

The [behind 1] remark tells us that "master" has received new changes on the remote. We must update "master" before we can integrate our own changes.

If properly configured, a plain "git pull" should suffice (after making "master" our active branch):

$ git checkout master
$ git pull

The last thing to check before actually starting the merge process is our current HEAD branch: we need to make sure that we've checked out the branch that should receivethe changes.

But since we just updated "master" and already performed a "git checkout master", we're good to go!

Starting the Merge

With such perfect preparation, the actual merge process itself is easy as pie:

$ git merge contact-form

Looking at our project's commit history, we'll notice that a new commit was created: a so-called "merge commit" that represents the actual "melting knot" that combines the two branches.

Our integration was successful and, if our feature work on "contact-form" is finished, we could safely delete that branch.

Dealing with Conflicts

Git will do what it can to make merging as easy as in our example. And in many cases, a merge will indeed be a walk in the park.

In some cases, however, the integration will not go as smoothly: if the branches contain incompatible changes, you will have to face (and solve) a "merge conflict". If you want to learn more about how to handle such a situation, have a look at Dealing with Merge Conflicts.

GIT: Submodules

Often in a project, you want to include libraries and other resources. The manual way is to simply download the necessary code files, copy them to your project, and commit the new files into your Git repository.

While this is a valid approach, it's not the cleanest one. By casually throwing those library files into your project, we're inviting a couple of problems:

• This mixes external code with our own, unique project files. The library, actually, is a project of itself and should be kept separate from our work. There's no need to keep these files in the same version control context as our project.
• Should the library change (because bugs were fixed or new features added), we'll have a hard time updating the library code. Again, we need to download the raw files and replace the original items.

Since these are quite common problems in everyday projects, Git of course offers a solution: Submodules.

Repositories Inside Repositories

A "Submodule" is just a standard Git repository. The only specialty is that it is nestedinside a parent repository. In the common case of including a code library, you can simply add the library as a Submodule in your main project.

A Submodule remains a fully functional Git repository: you can modify files, commit, pull, push, etc. from inside it like with any other repository.

Let's see how to work with Submodules in practice.

Adding a Submodule

In our sample project, we create a new "lib" folder to host this (and future) library code.

$ mkdir lib
$ cd lib

With the "git submodule add" command, we'll add a little Javascript library from GitHub:

$ git submodule add https://github.com/djyde/ToProgress

Let's have a look at what just happened:

• (1) The command started a simple cloning process of the specified Git repository:

Cloning into 'lib/ToProgress'...
remote: Counting objects: 180, done.
remote: Compressing objects: 100% (89/89), done.
remote: Total 180 (delta 51), reused 0 (delta 0), pack-reused 91
Receiving objects: 100% (180/180), 29.99 KiB | 0 bytes/s, done.
Resolving deltas: 100% (90/90), done.
Checking connectivity... done.

• (2) Of course, this is reflected in our file structure: our project now contains a new "ToProgess" folder inside the "lib" directory. As you can see from the ".git" subfolder contained herein, this is a fully-featured Git repository.

  

CONCEPT

It's important to understand that the actual contents of a Submodule are notstored in its parent repository. Only its remote URL, the local path inside the main project and the checked out revision are stored by the main repository.

Of course, the Submodule's working files are placed inside the specified directory in your project - in the end, you want to use the library's files! But they are not part of the parent project's version control contents.

• (3) A new ".gitmodules" file was created. This is where Git keeps track of our Submodules and their configuration:

[submodule "lib/ToProgress"]
    path = lib/ToProgress
    url = https://github.com/djyde/ToProgress

• (4) In case you're interested in the inner workings of Git: besides the ".gitmodules" configuration file, Git also keeps record of the Submodule in your local ".git/config" file. Finally, it also keeps a copy of each Submodule's .git repository in its internal ".git/modules" folder.

CONCEPT

Git's internal management of Submodules is quite complex (as you can already guess from all the .gitmodules, .git/config, and .git/modules entries...). Therefore, it's highly recommended not to mess with configuration files and values manually. Please do yourself a favor and always use proper Git commands to manage Submodules.

Let's have a look at our project's status:

$ git status
On branch master
Changes to be committed:
    (use "git reset HEAD ..." to unstage)

    new file:   .gitmodules
    new file:   lib/ToProgress

Git regards adding a Submodule as a modification like any other - and requests you to commit it to the repository:

$ git commit -m "Add 'ToProgress' Javascript library as Submodule"

Congratulations: we've now successfully added a Submodule to our main project! Before we look at a couple of use cases, let's see how you can clone a project that already has Submodules added.

Cloning a Project with Submodules

You already know that a project repository does not contain its Submodules' files; the parent repository only saves the Submodules' configurations as part of version control.
This shows when you clone a project that contains Submodules: by default, the "git clone" command only downloads the project itself. Our "lib" folder, however, would stay empty.

You have two options to end up with a populated "lib" folder (or wherever else you choose to save your Submodules; "lib" is just an example):

• (a) You can add the "--recurse-submodules" option to "git clone"; this tells Git to also initialize all Submodules when the cloning is finished.
• (b) If you used a simple "git clone" command without this option, you need to initialize the Submodules afterwards with "git submodule update --init --recursive"

Checking Out a Revision

A Git repository can have countless committed versions, but only one version's files can be in your working directory. Therefore, like with any Git repository, you have to decide which revision of your Submodule shall be checked out.

CONCEPT

Unlike normal Git repositories, Submodules always point to a specific commit - not a branch. This is because the contents of a branch can change over time, as new commits arrive. Pointing at a specific revision, on the other hand, guarantees that the correct code is always present.

Let's say we want to have an older version of our "ToProgress" library in our project. First, we'll have a look at the library's commit history. We change into the Submodule's base folder and call the "log" command:

$ cd lib/ToProgress/
$ git log --oneline --decorate

Before we take a look at the actual history, I'd like to stress an important point: Git commands are context-sensitive! By moving into the Submodule directory on the command line, all Git commands that we perform will be executed in the context of the Submodule, not its parent repository.

Now, in the log output, we spot a commit that is tagged "0.1.1":

83298f7 (HEAD, master) update .gitignore
a3b6186 remove page
ed693b7 update doc
3557a0e (tag: 0.1.1) change version code
2421796 update readme

This is the version we want to have in our project. To start with, we can simply check out this commit:

$ git checkout 0.1.1

Let's see what our parent repository thinks about all this. In the main project's base folder, execute:

$ git submodule status
+3557a0e0f7280fb3aba18fb9035d204c7de6344f   lib/ToProgress (0.1.1)

With "git submodule status", we're told which revision each Submodule is checked out at. The little "+" symbol in front of the hash is especially important: it tells us that the Submodule is at a different revision than is officially recorded in the parent repository. This makes sense - since we just changed the checked out revision to the commit tagged "0.1.1".

When performing a simple "git status" in the parent repository, we see that Git regards moving the Submodule's pointer as a change like any other:

$ git status
On branch master
Changes not staged for commit:
    (use "git add ..." to update what will be committed)
    (use "git checkout -- ..." to discard changes in working directory)

    modified:   lib/ToProgress (new commits)

We need to commit this to the repository in order to make it official:

$ git commit -a -m "Moved Submodule pointer to version 1.1.0"

Updating a Submodule When its Pointer was Moved

We just saw how to check out a Submodule at a specific revision. But what if one of our teammates does this in our project? Let's say we integrate his changes (through pull, merge, or rebase for example) after he has moved the Submodule pointer to a different revision:

$ git pull
Updating 43d0c47..3919c52
Fast-forward
 lib/ToProgress | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Git informs us, in a rather shy way, that "lib/ToProgress" was changed. Again, "git submodule status" provides more detailed information:

$ git submodule status
+83298f72c975c29f727c846579c297938492b245 lib/ToProgress (0.1.1-8-g83298f7)

Remember that little "+" sign? It tells us that the Submodule revision was moved - the version we currently have checked out in our project is not the one that is "officially" committed.

The "update" command helps us correct this:

$ git submodule update lib/ToProgress
Submodule path 'lib/ToProgress': checked out '3557a0e0f7280fb3aba18fb9035d204c7de6344f'

NOTE

In most cases, you can use the "git submodule" family of commands without specifying a particular Submodule. By providing a path like in the example above, however, you can address just a certain Submodule.

We now have the same version of the Submodule checked out that our teammate had committed to the repository.

Note that the "update" command also downloads changes for you: imagine that your teammate moved the Submodule's pointer to a revision that you don't have, yet. In that case, Git fetches the corresponding revision in the Submodule and then checks it out for you. Very handy.

Checking for New Changes in the Submodule

Normally, you don't want library code to change very often: you'll want to use a version of the Submodule that you've tested and which you know works flawlessly with your own code.
However, one of the best things about Submodules is that you can easily keep up with new releases (or minor new improvements).

Let's see if there's new code available in the Submodule:

    $ cd lib/ToProgress
    $ git fetch
    remote: Counting objects: 3, done.
    remote: Compressing objects: 100% (3/3), done.
    remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
    Unpacking objects: 100% (3/3), done.
    From https://github.com/djyde/ToProgress
        83298f7..3e20bc2  master     -> origin/master

Note that, to do this, we simply change into the Submodule folder - and can then work like with any normal Git repository (because it is a normal Git repository).
The "git fetch" command, in this case, shows that there are indeed some new changes on the Submodule's remote.

CONCEPT

Before we go ahead and integrate these changes, I'd like to stress an important point once more. When checking the Submodule's status, we're informed that we're on a detached HEAD:

$ git status
    HEAD detached at 3557a0e
    nothing to commit, working directory clean

Normally, in Git, you always have a certain branch checked out. However, you can also choose to check out a specific commit (one that is not the tip of a branch). This is a rather rare case in Git and should normally be avoided.

However, when working with Submodules, it is the normal state to have a certain commit (and not a branch) checked out. You want to make sure you have an exact, static commit checked out in your project - not a branch which, by its nature, moves on with newer commits.

Now, let's integrate the new changes by pulling them into our local Submodule repository. Note that you cannot use the shorthand "git pull" syntax but instead need to specify the remote and branch, too.

This is because of the "detached HEAD" state we're in: since you're not a local branch at the moment, you need to tell Git on which branch you want to integrate the pulled down changes.

$ git pull origin master

If you now were to execute "git status" once more, you'd notice that we're still on that same detached HEAD commit as before - the currently checked out commit was not moved like when we're on a branch. If we want to use the new Submodule code in our main project, we have to explicitly move the HEAD pointer:

$ git checkout master

We're done working in our Submodule; let's move back into our main project:

$ cd ../..
$ git submodule status
+3e20bc25457aa56bdb243c0e5c77549ea0a6a927 lib/ToProgress (0.1.1-9-g3e20bc2)

Since we've just moved the Submodule pointer to a different revision, we need to commit this change to the main repository to make it official.

Working in a Submodule

In some cases, you might want to make some custom changes to a Submodule. You've already seen that working in a Submodule is like working in any other Git repository: any Git commands that you perform inside a Submodule directory are executed in the context of that sub-repository.

Let's say you want to change a tiny bit in a Submodule; you make your changes in the corresponding files, add them to the staging area and commit them.

This might already be the first banana skin: you should make sure you currently have a branch checked out in the Submodule before you commit. That's because if you're in a detached HEAD situation, your commit will easily get lost: it's not attached to any branch and will be gone as soon as you check out anything else.

Apart from that, everything else you've already learned still applies: in the main project, "git submodule status" will tell you that the Submodule pointer was moved and that you'll have to commit the move.

By the way: In case you have uncommitted local changes inside the Submodule, Git will also tell you in the main project:

$ git status
...
    modified:   lib/ToProgress (modified content)

Make sure to always keep a clean state in your Submodules.

Deleting a Submodule

Rather seldomly will you want to remove a Submodule from your project. But if you really want to do this, please don't do this manually: trying to mess with all the configuration files in a correct way will almost inevitably cause problems.

$ git submodule deinit lib/ToProgress
$ git rm lib/ToPogress
$ git status
...
    modified:   .gitmodules
    deleted:    lib/ToProgress

With "git submodule deinit", we made sure that the Submodule is cleanly removed from the configuration files.
With "git rm", we finally delete the actual Submodule files - and other obsolete parts of your configuration.

Commit this and your Submodule will be cleanly removed from the project.

GIT: What's the difference between git fetch and git pull?

Before we talk about the differences between these two commands, let's stress their similarities: both are used to download new data from a remote repository.

Downloading data is an essential step in your daily work - because the remote data you are looking at in your local repository is just a "snapshot". It's only as up-to-date as the last time you explicitly downloaded fresh data from the remote with "fetch" or "pull". It's vital to keep this fact in mind when inspecting remote branches and commits!

Let's now look at the fine but important differences between "fetch" and "pull".

Fetch

$ git fetch origin

git fetch really only downloads new data from a remote repository - but it doesn't integrate any of this new data into your working files. Fetch is great for getting a fresh view on all the things that happened in a remote repository.
Due to it's "harmless" nature, you can rest assured: fetch will never manipulate, destroy, or screw up anything. This means you can never fetch often enough.

Pull

$ git pull origin master

git pull, in contrast, is used with a different goal in mind: to update your current HEAD branch with the latest changes from the remote server. This means that pull not only downloads new data; it also directly integrates it into your current working copy files. This has a couple of consequences:

• Since "git pull" tries to merge remote changes with your local ones, a so-called "merge conflict" can occur. Check out our in-depth tutorial on How to deal with merge conflicts for more information.
• Like for many other actions, it's highly recommended to start a "git pull" only with a clean working copy. This means that you should not have any uncommitted local changes before you pull. Use Git's Stash feature to save your local changes temporarily.

GIT: How to track a remote branch?

git fetch -all

git Checkout <branch_name>


Note: the command is case sensitive.

GIT: Updating your remote-tracking branches with git fetch

34.1. Fetch

The git fetch command updates your remote-tracking branches, i.e., it updates the local copy of branches stored in a remote repository. The following command updates the remote-tracking branches from the repository called origin.

git fetch origin

The fetch command only updates the remote-tracking branches and none of the local branches. It also does not change the working tree of the Git repository. Therefore, you can run the git fetch command at any point in time.

After reviewing the changes in the remote tracking branchm you can merge the changes into your local branches or rebase your local branches onto the remote-tracking branch.

Alternatively you can also use the git cherry-pick commit_id command to take over only selected commits.

34.2. Fetch from all remote repositories

The git fetch command updates only the remote-tracking branches for one remote repository. In case you want to update the remote-tracking branches of all your remote repositories you can use the following command.

# simplification of the fetch command
# this runs git fetch for every remote repository
git remote update

# the same but remove all stale branches which
# are not in the remote anymore
git remote update --prune

34.3. Compare remote-tracking branch with local branch

The following code shows a few options how you can compare your branches.

# show the log entries between the last local commit and the
# remote branch
git log HEAD..origin/master

# show the diff for each patch
git log -p HEAD..origin/master

# show a single diff
git diff HEAD...origin/master

# instead of using HEAD you can also
# specify the branches directly
git diff master origin/master

The above commands show the changes introduced in HEAD compared to origin. If you want to see the changes in origin compared to HEAD, you can switch the arguments or use the -R parameter.

34.4. Rebase your local branch onto the remote-tracking branch

You can rebase your current local branch onto a remote-tracking branch. The following commands demonstrate that.

# assume you want to rebase master based on the latest fetch
# therefore check it out
git checkout master

# update your remote-tracking branch
git fetch

# rebase your master onto origin/master
git rebase origin/master

More information on the rebase command can be found in Rebasing branches.

34.5. Fetch compared with pull

The git pull command performs a git fetch and git merge (or git rebase based on your Git settings). The git fetch does not perform any operations on your local branches. You can always run the fetch command and review the incoming changes.