Skip to content
Navigation Menu
Toggle navigation
Sign in
In this repository
All GitHub Enterprise
↵
Jump to
↵
No suggested jump to results
In this repository
All GitHub Enterprise
↵
Jump to
↵
In this organization
All GitHub Enterprise
↵
Jump to
↵
In this repository
All GitHub Enterprise
↵
Jump to
↵
Sign in
Reseting focus
You signed in with another tab or window.
Reload
to refresh your session.
You signed out in another tab or window.
Reload
to refresh your session.
You switched accounts on another tab or window.
Reload
to refresh your session.
Dismiss alert
{{ message }}
git-mirror
/
git
Public
Notifications
You must be signed in to change notification settings
Fork
0
Star
0
Code
Issues
0
Pull requests
0
Actions
Projects
0
Security
Insights
Additional navigation options
Code
Issues
Pull requests
Actions
Projects
Security
Insights
Files
a5d86f7
Documentation
howto
technical
.gitignore
api-allocation-growing.txt
api-builtin.txt
api-decorate.txt
api-diff.txt
api-directory-listing.txt
api-gitattributes.txt
api-grep.txt
api-hash.txt
api-in-core-index.txt
api-index-skel.txt
api-index.sh
api-lockfile.txt
api-object-access.txt
api-parse-options.txt
api-path-list.txt
api-quote.txt
api-revision-walking.txt
api-run-command.txt
api-setup.txt
api-strbuf.txt
api-tree-walking.txt
api-xdiff-interface.txt
pack-format.txt
pack-heuristics.txt
pack-protocol.txt
racy-git.txt
send-pack-pipeline.txt
shallow.txt
trivial-merge.txt
.gitignore
CodingGuidelines
Makefile
RelNotes-1.5.0.1.txt
RelNotes-1.5.0.2.txt
RelNotes-1.5.0.3.txt
RelNotes-1.5.0.4.txt
RelNotes-1.5.0.5.txt
RelNotes-1.5.0.6.txt
RelNotes-1.5.0.7.txt
RelNotes-1.5.0.txt
RelNotes-1.5.1.1.txt
RelNotes-1.5.1.2.txt
RelNotes-1.5.1.3.txt
RelNotes-1.5.1.4.txt
RelNotes-1.5.1.5.txt
RelNotes-1.5.1.6.txt
RelNotes-1.5.1.txt
RelNotes-1.5.2.1.txt
RelNotes-1.5.2.2.txt
RelNotes-1.5.2.3.txt
RelNotes-1.5.2.4.txt
RelNotes-1.5.2.5.txt
RelNotes-1.5.2.txt
RelNotes-1.5.3.1.txt
RelNotes-1.5.3.2.txt
RelNotes-1.5.3.3.txt
RelNotes-1.5.3.4.txt
RelNotes-1.5.3.5.txt
RelNotes-1.5.3.6.txt
RelNotes-1.5.3.7.txt
RelNotes-1.5.3.8.txt
RelNotes-1.5.3.txt
RelNotes-1.5.4.txt
SubmittingPatches
asciidoc.conf
blame-options.txt
build-docdep.perl
callouts.xsl
cat-texi.perl
cmd-list.perl
config.txt
core-tutorial.txt
cvs-migration.txt
diff-format.txt
diff-generate-patch.txt
diff-options.txt
diffcore.txt
docbook-xsl.css
docbook.xsl
everyday.txt
fetch-options.txt
fix-texi.perl
git-add.txt
git-am.txt
git-annotate.txt
git-apply.txt
git-archimport.txt
git-archive.txt
git-bisect.txt
git-blame.txt
git-branch.txt
git-bundle.txt
git-cat-file.txt
git-check-attr.txt
git-check-ref-format.txt
git-checkout-index.txt
git-checkout.txt
git-cherry-pick.txt
git-cherry.txt
git-citool.txt
git-clean.txt
git-clone.txt
git-commit-tree.txt
git-commit.txt
git-config.txt
git-count-objects.txt
git-cvsexportcommit.txt
git-cvsimport.txt
git-cvsserver.txt
git-daemon.txt
git-describe.txt
git-diff-files.txt
git-diff-index.txt
git-diff-tree.txt
git-diff.txt
git-fast-export.txt
git-fast-import.txt
git-fetch-pack.txt
git-fetch.txt
git-filter-branch.txt
git-fmt-merge-msg.txt
git-for-each-ref.txt
git-format-patch.txt
git-fsck-objects.txt
git-fsck.txt
git-gc.txt
git-get-tar-commit-id.txt
git-grep.txt
git-gui.txt
git-hash-object.txt
git-help.txt
git-http-fetch.txt
git-http-push.txt
git-imap-send.txt
git-index-pack.txt
git-init-db.txt
git-init.txt
git-instaweb.txt
git-log.txt
git-lost-found.txt
git-ls-files.txt
git-ls-remote.txt
git-ls-tree.txt
git-mailinfo.txt
git-mailsplit.txt
git-merge-base.txt
git-merge-file.txt
git-merge-index.txt
git-merge-one-file.txt
git-merge-tree.txt
git-merge.txt
git-mergetool.txt
git-mktag.txt
git-mktree.txt
git-mv.txt
git-name-rev.txt
git-pack-objects.txt
git-pack-redundant.txt
git-pack-refs.txt
git-parse-remote.txt
git-patch-id.txt
git-peek-remote.txt
git-prune-packed.txt
git-prune.txt
git-pull.txt
git-push.txt
git-quiltimport.txt
git-read-tree.txt
git-rebase.txt
git-receive-pack.txt
git-reflog.txt
git-relink.txt
git-remote.txt
git-repack.txt
git-repo-config.txt
git-request-pull.txt
git-rerere.txt
git-reset.txt
git-rev-list.txt
git-rev-parse.txt
git-revert.txt
git-rm.txt
git-send-email.txt
git-send-pack.txt
git-sh-setup.txt
git-shell.txt
git-shortlog.txt
git-show-branch.txt
git-show-index.txt
git-show-ref.txt
git-show.txt
git-stash.txt
git-status.txt
git-stripspace.txt
git-submodule.txt
git-svn.txt
git-symbolic-ref.txt
git-tag.txt
git-tar-tree.txt
git-tools.txt
git-unpack-file.txt
git-unpack-objects.txt
git-update-index.txt
git-update-ref.txt
git-update-server-info.txt
git-upload-archive.txt
git-upload-pack.txt
git-var.txt
git-verify-pack.txt
git-verify-tag.txt
git-whatchanged.txt
git-write-tree.txt
git.txt
gitattributes.txt
gitcli.txt
gitignore.txt
gitk.txt
gitmodules.txt
glossary.txt
hooks.txt
howto-index.sh
i18n.txt
install-doc-quick.sh
install-webdoc.sh
merge-options.txt
merge-strategies.txt
pretty-formats.txt
pretty-options.txt
pull-fetch-param.txt
repository-layout.txt
rev-list-options.txt
tutorial-2.txt
tutorial.txt
urls-remotes.txt
urls.txt
user-manual.conf
user-manual.txt
arm
compat
contrib
git-gui
gitk-git
gitweb
mozilla-sha1
perl
ppc
t
templates
xdiff
.gitignore
.mailmap
COPYING
GIT-VERSION-GEN
INSTALL
Makefile
README
RelNotes
alloc.c
archive-tar.c
archive-zip.c
archive.c
archive.h
attr.c
attr.h
base85.c
blob.c
blob.h
builtin-add.c
builtin-annotate.c
builtin-apply.c
builtin-archive.c
builtin-blame.c
builtin-branch.c
builtin-bundle.c
builtin-cat-file.c
builtin-check-attr.c
builtin-check-ref-format.c
builtin-checkout-index.c
builtin-clean.c
builtin-commit-tree.c
builtin-commit.c
builtin-config.c
builtin-count-objects.c
builtin-describe.c
builtin-diff-files.c
builtin-diff-index.c
builtin-diff-tree.c
builtin-diff.c
builtin-fast-export.c
builtin-fetch--tool.c
builtin-fetch-pack.c
builtin-fetch.c
builtin-fmt-merge-msg.c
builtin-for-each-ref.c
builtin-fsck.c
builtin-gc.c
builtin-grep.c
builtin-http-fetch.c
builtin-init-db.c
builtin-log.c
builtin-ls-files.c
builtin-ls-remote.c
builtin-ls-tree.c
builtin-mailinfo.c
builtin-mailsplit.c
builtin-merge-base.c
builtin-merge-file.c
builtin-merge-ours.c
builtin-mv.c
builtin-name-rev.c
builtin-pack-objects.c
builtin-pack-refs.c
builtin-prune-packed.c
builtin-prune.c
builtin-push.c
builtin-read-tree.c
builtin-reflog.c
builtin-rerere.c
builtin-reset.c
builtin-rev-list.c
builtin-rev-parse.c
builtin-revert.c
builtin-rm.c
builtin-send-pack.c
builtin-shortlog.c
builtin-show-branch.c
builtin-show-ref.c
builtin-stripspace.c
builtin-symbolic-ref.c
builtin-tag.c
builtin-tar-tree.c
builtin-unpack-objects.c
builtin-update-index.c
builtin-update-ref.c
builtin-upload-archive.c
builtin-verify-pack.c
builtin-verify-tag.c
builtin-write-tree.c
builtin.h
bundle.c
bundle.h
cache-tree.c
cache-tree.h
cache.h
check-builtins.sh
check-racy.c
color.c
color.h
combine-diff.c
command-list.txt
commit.c
commit.h
config.c
config.mak.in
configure.ac
connect.c
convert.c
copy.c
csum-file.c
csum-file.h
ctype.c
daemon.c
date.c
decorate.c
decorate.h
delta.h
diff-delta.c
diff-lib.c
diff.c
diff.h
diffcore-break.c
diffcore-delta.c
diffcore-order.c
diffcore-pickaxe.c
diffcore-rename.c
diffcore.h
dir.c
dir.h
dump-cache-tree.c
entry.c
environment.c
exec_cmd.c
exec_cmd.h
fast-import.c
fetch-pack.h
fixup-builtins
generate-cmdlist.sh
git-add--interactive.perl
git-am.sh
git-archimport.perl
git-bisect.sh
git-checkout.sh
git-clone.sh
git-compat-util.h
git-cvsexportcommit.perl
git-cvsimport.perl
git-cvsserver.perl
git-filter-branch.sh
git-help--browse.sh
git-instaweb.sh
git-lost-found.sh
git-merge-octopus.sh
git-merge-one-file.sh
git-merge-resolve.sh
git-merge-stupid.sh
git-merge.sh
git-mergetool.sh
git-parse-remote.sh
git-pull.sh
git-quiltimport.sh
git-rebase--interactive.sh
git-rebase.sh
git-relink.perl
git-remote.perl
git-repack.sh
git-request-pull.sh
git-send-email.perl
git-sh-setup.sh
git-stash.sh
git-submodule.sh
git-svn.perl
git.c
git.spec.in
grep.c
grep.h
hash-object.c
hash.c
hash.h
help.c
http-push.c
http-walker.c
http.c
http.h
ident.c
imap-send.c
index-pack.c
interpolate.c
interpolate.h
list-objects.c
list-objects.h
lockfile.c
log-tree.c
log-tree.h
mailmap.c
mailmap.h
match-trees.c
merge-file.c
merge-index.c
merge-recursive.c
merge-tree.c
mktag.c
mktree.c
object-refs.c
object.c
object.h
pack-check.c
pack-redundant.c
pack-write.c
pack.h
pager.c
parse-options.c
parse-options.h
patch-delta.c
patch-id.c
patch-ids.c
patch-ids.h
path-list.c
path-list.h
path.c
pkt-line.c
pkt-line.h
pretty.c
progress.c
progress.h
quote.c
quote.h
reachable.c
reachable.h
read-cache.c
receive-pack.c
reflog-walk.c
reflog-walk.h
refs.c
refs.h
remote.c
remote.h
revision.c
revision.h
run-command.c
run-command.h
send-pack.h
server-info.c
setup.c
sha1_file.c
sha1_name.c
shallow.c
shell.c
show-index.c
sideband.c
sideband.h
strbuf.c
strbuf.h
symlinks.c
tag.c
tag.h
tar.h
test-absolute-path.c
test-chmtime.c
test-date.c
test-delta.c
test-genrandom.c
test-match-trees.c
test-parse-options.c
test-sha1.c
test-sha1.sh
trace.c
transport.c
transport.h
tree-diff.c
tree-walk.c
tree-walk.h
tree.c
tree.h
unpack-file.c
unpack-trees.c
unpack-trees.h
update-server-info.c
upload-pack.c
usage.c
utf8.c
utf8.h
var.c
walker.c
walker.h
write_or_die.c
ws.c
wt-status.c
wt-status.h
xdiff-interface.c
xdiff-interface.h
Breadcrumbs
git
/
Documentation
/
technical
/
racy-git.txt
Blame
Blame
Latest commit
History
History
195 lines (161 loc) · 8.46 KB
Breadcrumbs
git
/
Documentation
/
technical
/
racy-git.txt
Top
File metadata and controls
Code
Blame
195 lines (161 loc) · 8.46 KB
Raw
Use of index and Racy git problem ================================= Background ---------- The index is one of the most important data structures in git. It represents a virtual working tree state by recording list of paths and their object names and serves as a staging area to write out the next tree object to be committed. The state is "virtual" in the sense that it does not necessarily have to, and often does not, match the files in the working tree. There are cases git needs to examine the differences between the virtual working tree state in the index and the files in the working tree. The most obvious case is when the user asks `git diff` (or its low level implementation, `git diff-files`) or `git-ls-files --modified`. In addition, git internally checks if the files in the working tree are different from what are recorded in the index to avoid stomping on local changes in them during patch application, switching branches, and merging. In order to speed up this comparison between the files in the working tree and the index entries, the index entries record the information obtained from the filesystem via `lstat(2)` system call when they were last updated. When checking if they differ, git first runs `lstat(2)` on the files and compares the result with this information (this is what was originally done by the `ce_match_stat()` function, but the current code does it in `ce_match_stat_basic()` function). If some of these "cached stat information" fields do not match, git can tell that the files are modified without even looking at their contents. Note: not all members in `struct stat` obtained via `lstat(2)` are used for this comparison. For example, `st_atime` obviously is not useful. Currently, git compares the file type (regular files vs symbolic links) and executable bits (only for regular files) from `st_mode` member, `st_mtime` and `st_ctime` timestamps, `st_uid`, `st_gid`, `st_ino`, and `st_size` members. With a `USE_STDEV` compile-time option, `st_dev` is also compared, but this is not enabled by default because this member is not stable on network filesystems. With `USE_NSEC` compile-time option, `st_mtim.tv_nsec` and `st_ctim.tv_nsec` members are also compared, but this is not enabled by default because the value of this member becomes meaningless once the inode is evicted from the inode cache on filesystems that do not store it on disk. Racy git -------- There is one slight problem with the optimization based on the cached stat information. Consider this sequence: : modify 'foo' $ git update-index 'foo' : modify 'foo' again, in-place, without changing its size The first `update-index` computes the object name of the contents of file `foo` and updates the index entry for `foo` along with the `struct stat` information. If the modification that follows it happens very fast so that the file's `st_mtime` timestamp does not change, after this sequence, the cached stat information the index entry records still exactly match what you would see in the filesystem, even though the file `foo` is now different. This way, git can incorrectly think files in the working tree are unmodified even though they actually are. This is called the "racy git" problem (discovered by Pasky), and the entries that appear clean when they may not be because of this problem are called "racily clean". To avoid this problem, git does two things: . When the cached stat information says the file has not been modified, and the `st_mtime` is the same as (or newer than) the timestamp of the index file itself (which is the time `git update-index foo` finished running in the above example), it also compares the contents with the object registered in the index entry to make sure they match. . When the index file is updated that contains racily clean entries, cached `st_size` information is truncated to zero before writing a new version of the index file. Because the index file itself is written after collecting all the stat information from updated paths, `st_mtime` timestamp of it is usually the same as or newer than any of the paths the index contains. And no matter how quick the modification that follows `git update-index foo` finishes, the resulting `st_mtime` timestamp on `foo` cannot get a value earlier than the index file. Therefore, index entries that can be racily clean are limited to the ones that have the same timestamp as the index file itself. The callers that want to check if an index entry matches the corresponding file in the working tree continue to call `ce_match_stat()`, but with this change, `ce_match_stat()` uses `ce_modified_check_fs()` to see if racily clean ones are actually clean after comparing the cached stat information using `ce_match_stat_basic()`. The problem the latter solves is this sequence: $ git update-index 'foo' : modify 'foo' in-place without changing its size : wait for enough time $ git update-index 'bar' Without the latter, the timestamp of the index file gets a newer value, and falsely clean entry `foo` would not be caught by the timestamp comparison check done with the former logic anymore. The latter makes sure that the cached stat information for `foo` would never match with the file in the working tree, so later checks by `ce_match_stat_basic()` would report that the index entry does not match the file and git does not have to fall back on more expensive `ce_modified_check_fs()`. Runtime penalty --------------- The runtime penalty of falling back to `ce_modified_check_fs()` from `ce_match_stat()` can be very expensive when there are many racily clean entries. An obvious way to artificially create this situation is to give the same timestamp to all the files in the working tree in a large project, run `git update-index` on them, and give the same timestamp to the index file: $ date >.datestamp $ git ls-files | xargs touch -r .datestamp $ git ls-files | git update-index --stdin $ touch -r .datestamp .git/index This will make all index entries racily clean. The linux-2.6 project, for example, there are over 20,000 files in the working tree. On my Athron 64X2 3800+, after the above: $ /usr/bin/time git diff-files 1.68user 0.54system 0:02.22elapsed 100%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+67111minor)pagefaults 0swaps $ git update-index MAINTAINERS $ /usr/bin/time git diff-files 0.02user 0.12system 0:00.14elapsed 100%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+935minor)pagefaults 0swaps Running `git update-index` in the middle checked the racily clean entries, and left the cached `st_mtime` for all the paths intact because they were actually clean (so this step took about the same amount of time as the first `git diff-files`). After that, they are not racily clean anymore but are truly clean, so the second invocation of `git diff-files` fully took advantage of the cached stat information. Avoiding runtime penalty ------------------------ In order to avoid the above runtime penalty, post 1.4.2 git used to have a code that made sure the index file got timestamp newer than the youngest files in the index when there are many young files with the same timestamp as the resulting index file would otherwise would have by waiting before finishing writing the index file out. I suspected that in practice the situation where many paths in the index are all racily clean was quite rare. The only code paths that can record recent timestamp for large number of paths are: . Initial `git add .` of a large project. . `git checkout` of a large project from an empty index into an unpopulated working tree. Note: switching branches with `git checkout` keeps the cached stat information of existing working tree files that are the same between the current branch and the new branch, which are all older than the resulting index file, and they will not become racily clean. Only the files that are actually checked out can become racily clean. In a large project where raciness avoidance cost really matters, however, the initial computation of all object names in the index takes more than one second, and the index file is written out after all that happens. Therefore the timestamp of the index file will be more than one seconds later than the youngest file in the working tree. This means that in these cases there actually will not be any racily clean entry in the resulting index. Based on this discussion, the current code does not use the "workaround" to avoid the runtime penalty that does not exist in practice anymore. This was done with commit 0fc82cff on Aug 15, 2006.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
You can’t perform that action at this time.