Merge branch 'jn/merge-renormalize'

* jn/merge-renormalize:
  merge-recursive --renormalize
  rerere: never renormalize
  rerere: migrate to parse-options API
  t4200 (rerere): modernize style
  ll-merge: let caller decide whether to renormalize
  ll-merge: make flag easier to populate
  Documentation/technical: document ll_merge
  merge-trees: let caller decide whether to renormalize
  merge-trees: push choice to renormalize away from low level
  t6038 (merge.renormalize): check that it can be turned off
  t6038 (merge.renormalize): try checkout -m and cherry-pick
  t6038 (merge.renormalize): style nitpicks
  Don't expand CRLFs when normalizing text during merge
  Try normalizing files to avoid delete/modify conflicts when merging
  Avoid conflicts when merging branches with mixed normalization

Conflicts:
	builtin/rerere.c
	t/t4200-rerere.sh

Documentation/gitattributes.txt

@@ -317,6 +317,17 @@ command is "cat").
 	smudge = cat
 ------------------------
 
+For best results, `clean` should not alter its output further if it is
+run twice ("clean->clean" should be equivalent to "clean"), and
+multiple `smudge` commands should not alter `clean`'s output
+("smudge->smudge->clean" should be equivalent to "clean").  See the
+section on merging below.
+
+The "indent" filter is well-behaved in this regard: it will not modify
+input that is already correctly indented.  In this case, the lack of a
+smudge filter means that the clean filter _must_ accept its own output
+without modifying it.
+
 
 Interaction between checkin/checkout attributes
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -331,6 +342,29 @@ In the check-out codepath, the blob content is first converted
 with `text`, and then `ident` and fed to `filter`.
 
 
+Merging branches with differing checkin/checkout attributes
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If you have added attributes to a file that cause the canonical
+repository format for that file to change, such as adding a
+clean/smudge filter or text/eol/ident attributes, merging anything
+where the attribute is not in place would normally cause merge
+conflicts.
+
+To prevent these unnecessary merge conflicts, git can be told to run a
+virtual check-out and check-in of all three stages of a file when
+resolving a three-way merge by setting the `merge.renormalize`
+configuration variable.  This prevents changes caused by check-in
+conversion from causing spurious merge conflicts when a converted file
+is merged with an unconverted file.
+
+As long as a "smudge->clean" results in the same output as a "clean"
+even on files that are already smudged, this strategy will
+automatically resolve all filter-related conflicts.  Filters that do
+not act in this way may cause additional merge conflicts that must be
+resolved manually.
+
+
 Generating diff text
 ~~~~~~~~~~~~~~~~~~~~
 

Documentation/merge-config.txt

@@ -15,6 +15,16 @@ merge.renameLimit::
 	during a merge; if not specified, defaults to the value of
 	diff.renameLimit.
 
+merge.renormalize::
+	Tell git that canonical representation of files in the
+	repository has changed over time (e.g. earlier commits record
+	text files with CRLF line endings, but recent ones use LF line
+	endings).  In such a repository, git can convert the data
+	recorded in commits to a canonical form before performing a
+	merge to reduce unnecessary conflicts.  For more information,
+	see section "Merging branches with differing checkin/checkout
+	attributes" in linkgit:gitattributes[5].
+
 merge.stat::
 	Whether to print the diffstat between ORIG_HEAD and the merge result
 	at the end of the merge.  True by default.

Documentation/merge-strategies.txt

@@ -40,6 +40,18 @@ the other tree did, declaring 'our' history contains all that happened in it.
 theirs;;
 	This is opposite of 'ours'.
 
+renormalize;;
+	This runs a virtual check-out and check-in of all three stages
+	of a file when resolving a three-way merge.  This option is
+	meant to be used when merging branches with different clean
+	filters or end-of-line normalization rules.  See "Merging
+	branches with differing checkin/checkout attributes" in
+	linkgit:gitattributes[5] for details.
+
+no-renormalize;;
+	Disables the `renormalize` option.  This overrides the
+	`merge.renormalize` configuration variable.
+
 subtree[=path];;
 	This option is a more advanced form of 'subtree' strategy, where
 	the strategy makes a guess on how two trees must be shifted to

Documentation/technical/api-merge.txt

@@ -0,0 +1,73 @@
+merge API
+=========
+
+The merge API helps a program to reconcile two competing sets of
+improvements to some files (e.g., unregistered changes from the work
+tree versus changes involved in switching to a new branch), reporting
+conflicts if found.  The library called through this API is
+responsible for a few things.
+
+ * determining which trees to merge (recursive ancestor consolidation);
+
+ * lining up corresponding files in the trees to be merged (rename
+   detection, subtree shifting), reporting edge cases like add/add
+   and rename/rename conflicts to the user;
+
+ * performing a three-way merge of corresponding files, taking
+   path-specific merge drivers (specified in `.gitattributes`)
+   into account.
+
+Low-level (single file) merge
+-----------------------------
+
+`ll_merge`::
+
+	Perform a three-way single-file merge in core.  This is
+	a thin wrapper around `xdl_merge` that takes the path and
+	any merge backend specified in `.gitattributes` or
+	`.git/info/attributes` into account.  Returns 0 for a
+	clean merge.
+
+The caller:
+
+1. allocates an mmbuffer_t variable for the result;
+2. allocates and fills variables with the file's original content
+   and two modified versions (using `read_mmfile`, for example);
+3. calls ll_merge();
+4. reads the output from result_buf.ptr and result_buf.size;
+5. releases buffers when finished (free(ancestor.ptr); free(ours.ptr);
+   free(theirs.ptr); free(result_buf.ptr);).
+
+If the modifications do not merge cleanly, `ll_merge` will return a
+nonzero value and `result_buf` will generally include a description of
+the conflict bracketed by markers such as the traditional `<<<<<<<`
+and `>>>>>>>`.
+
+The `ancestor_label`, `our_label`, and `their_label` parameters are
+used to label the different sides of a conflict if the merge driver
+supports this.
+
+The `flag` parameter is a bitfield:
+
+ - The `LL_OPT_VIRTUAL_ANCESTOR` bit indicates whether this is an
+   internal merge to consolidate ancestors for a recursive merge.
+
+ - The `LL_OPT_FAVOR_MASK` bits allow local conflicts to be automatically
+   resolved in favor of one side or the other (as in 'git merge-file'
+   `--ours`/`--theirs`/`--union`).
+   They can be populated by `create_ll_flag`, whose argument can be
+   `XDL_MERGE_FAVOR_OURS`, `XDL_MERGE_FAVOR_THEIRS`, or
+   `XDL_MERGE_FAVOR_UNION`.
+
+Everything else
+---------------
+
+Talk about <merge-recursive.h> and merge_file():
+
+ - merge_trees() to merge with rename detection
+ - merge_recursive() for ancestor consolidation
+ - try_merge_command() for other strategies
+ - conflict format
+ - merge options
+
+(Daniel, Miklos, Stephan, JC)

builtin/checkout.c

@@ -154,6 +154,10 @@ static int checkout_merged(int pos, struct checkout *state)
 	read_mmblob(&ours, active_cache[pos+1]->sha1);
 	read_mmblob(&theirs, active_cache[pos+2]->sha1);
 
+	/*
+	 * NEEDSWORK: re-create conflicts from merges with
+	 * merge.renormalize set, too
+	 */
 	status = ll_merge(&result_buf, path, &ancestor, "base",
 			  &ours, "ours", &theirs, "theirs", 0);
 	free(ancestor.ptr);
@@ -437,6 +441,13 @@ static int merge_working_tree(struct checkout_opts *opts,
 			 */
 
 			add_files_to_cache(NULL, NULL, 0);
+			/*
+			 * NEEDSWORK: carrying over local changes
+			 * when branches have different end-of-line
+			 * normalization (or clean+smudge rules) is
+			 * a pain; plumb in an option to set
+			 * o.renormalize?
+			 */
 			init_merge_options(&o);
 			o.verbosity = 0;
 			work = write_tree_from_memory(&o);

builtin/merge-recursive.c

@@ -48,6 +48,10 @@ int cmd_merge_recursive(int argc, const char **argv, const char *prefix)
 				o.subtree_shift = "";
 			else if (!prefixcmp(arg+2, "subtree="))
 				o.subtree_shift = arg + 10;
+			else if (!strcmp(arg+2, "renormalize"))
+				o.renormalize = 1;
+			else if (!strcmp(arg+2, "no-renormalize"))
+				o.renormalize = 0;
 			else
 				die("Unknown option %s", arg);
 			continue;

builtin/merge.c

@@ -54,6 +54,7 @@ static size_t use_strategies_nr, use_strategies_alloc;
 static const char **xopts;
 static size_t xopts_nr, xopts_alloc;
 static const char *branch;
+static int option_renormalize;
 static int verbosity;
 static int allow_rerere_auto;
 
@@ -504,6 +505,8 @@ static int git_merge_config(const char *k, const char *v, void *cb)
 		return git_config_string(&pull_octopus, k, v);
 	else if (!strcmp(k, "merge.log") || !strcmp(k, "merge.summary"))
 		option_log = git_config_bool(k, v);
+	else if (!strcmp(k, "merge.renormalize"))
+		option_renormalize = git_config_bool(k, v);
 	return git_diff_ui_config(k, v, cb);
 }
 
@@ -625,6 +628,11 @@ static int try_merge_strategy(const char *strategy, struct commit_list *common,
 		if (!strcmp(strategy, "subtree"))
 			o.subtree_shift = "";
 
+		o.renormalize = option_renormalize;
+
+		/*
+		 * NEEDSWORK: merge with table in builtin/merge-recursive
+		 */
 		for (x = 0; x < xopts_nr; x++) {
 			if (!strcmp(xopts[x], "ours"))
 				o.recursive_variant = MERGE_RECURSIVE_OURS;
@@ -634,6 +642,10 @@ static int try_merge_strategy(const char *strategy, struct commit_list *common,
 				o.subtree_shift = "";
 			else if (!prefixcmp(xopts[x], "subtree="))
 				o.subtree_shift = xopts[x]+8;
+			else if (!strcmp(xopts[x], "renormalize"))
+				o.renormalize = 1;
+			else if (!strcmp(xopts[x], "no-renormalize"))
+				o.renormalize = 0;
 			else
 				die("Unknown option for merge-recursive: -X%s", xopts[x]);
 		}
@@ -818,7 +830,7 @@ static int finish_automerge(struct commit_list *common,
 	return 0;
 }
 
-static int suggest_conflicts(void)
+static int suggest_conflicts(int renormalizing)
 {
 	FILE *fp;
 	int pos;
@@ -1303,5 +1315,5 @@ int cmd_merge(int argc, const char **argv, const char *prefix)
 			"stopped before committing as requested\n");
 		return 0;
 	} else
-		return suggest_conflicts();
+		return suggest_conflicts(option_renormalize);
 }

builtin/rerere.c

@@ -1,13 +1,16 @@
 #include "builtin.h"
 #include "cache.h"
 #include "dir.h"
+#include "parse-options.h"
 #include "string-list.h"
 #include "rerere.h"
 #include "xdiff/xdiff.h"
 #include "xdiff-interface.h"
 
-static const char git_rerere_usage[] =
-"git rerere [clear | status | diff | gc]";
+static const char * const rerere_usage[] = {
+	"git rerere [clear | status | diff | gc]",
+	NULL,
+};
 
 /* these values are days */
 static int cutoff_noresolve = 15;
@@ -114,52 +117,53 @@ static int diff_two(const char *file1, const char *label1,
 int cmd_rerere(int argc, const char **argv, const char *prefix)
 {
 	struct string_list merge_rr = STRING_LIST_INIT_DUP;
-	int i, fd, flags = 0;
-
-	if (2 < argc) {
-		if (!strcmp(argv[1], "-h"))
-			usage(git_rerere_usage);
-		if (!strcmp(argv[1], "--rerere-autoupdate"))
-			flags = RERERE_AUTOUPDATE;
-		else if (!strcmp(argv[1], "--no-rerere-autoupdate"))
-			flags = RERERE_NOAUTOUPDATE;
-		if (flags) {
-			argc--;
-			argv++;
-		}
-	}
-	if (argc < 2)
+	int i, fd, autoupdate = -1, flags = 0;
+
+	struct option options[] = {
+		OPT_SET_INT(0, "rerere-autoupdate", &autoupdate,
+			"register clean resolutions in index", 1),
+		OPT_END(),
+	};
+
+	argc = parse_options(argc, argv, prefix, options, rerere_usage, 0);
+
+	if (autoupdate == 1)
+		flags = RERERE_AUTOUPDATE;
+	if (autoupdate == 0)
+		flags = RERERE_NOAUTOUPDATE;
+
+	if (argc < 1)
 		return rerere(flags);
 
-	if (!strcmp(argv[1], "forget")) {
-		const char **pathspec = get_pathspec(prefix, argv + 2);
+	if (!strcmp(argv[0], "forget")) {
+		const char **pathspec = get_pathspec(prefix, argv + 1);
 		return rerere_forget(pathspec);
 	}
 
 	fd = setup_rerere(&merge_rr, flags);
 	if (fd < 0)
 		return 0;
 
-	if (!strcmp(argv[1], "clear")) {
+	if (!strcmp(argv[0], "clear")) {
 		for (i = 0; i < merge_rr.nr; i++) {
 			const char *name = (const char *)merge_rr.items[i].util;
 			if (!has_rerere_resolution(name))
 				unlink_rr_item(name);
 		}
 		unlink_or_warn(git_path("MERGE_RR"));
-	} else if (!strcmp(argv[1], "gc"))
+	} else if (!strcmp(argv[0], "gc"))
 		garbage_collect(&merge_rr);
-	else if (!strcmp(argv[1], "status"))
+	else if (!strcmp(argv[0], "status"))
 		for (i = 0; i < merge_rr.nr; i++)
 			printf("%s\n", merge_rr.items[i].string);
-	else if (!strcmp(argv[1], "diff"))
+	else if (!strcmp(argv[0], "diff"))
 		for (i = 0; i < merge_rr.nr; i++) {
 			const char *path = merge_rr.items[i].string;
 			const char *name = (const char *)merge_rr.items[i].util;
 			diff_two(rerere_path(name, "preimage"), path, path, path);
 		}
 	else
-		usage(git_rerere_usage);
+		usage_with_options(rerere_usage, options);
 
 	string_list_clear(&merge_rr, 1);
 	return 0;

builtin/revert.c

@@ -316,6 +316,13 @@ static int do_recursive_merge(struct commit *base, struct commit *next,
 	index_fd = hold_locked_index(&index_lock, 1);
 
 	read_cache();
+
+	/*
+	 * NEEDSWORK: cherry-picking between branches with
+	 * different end-of-line normalization is a pain;
+	 * plumb in an option to set o.renormalize?
+	 * (or better: arbitrary -X options)
+	 */
 	init_merge_options(&o);
 	o.ancestor = base ? base_label : "(empty tree)";
 	o.branch1 = "HEAD";

cache.h

@@ -1057,6 +1057,7 @@ extern void trace_argv_printf(const char **argv, const char *format, ...);
 extern int convert_to_git(const char *path, const char *src, size_t len,
                           struct strbuf *dst, enum safe_crlf checksafe);
 extern int convert_to_working_tree(const char *path, const char *src, size_t len, struct strbuf *dst);
+extern int renormalize_buffer(const char *path, const char *src, size_t len, struct strbuf *dst);
 
 /* add */
 /*