Implement file+noindex:///local/repositories

[phadej]

Resolve #6359

preferred-versions are left out for now.
It shouldn't be difficult to add, but needs work nevertheless.

Also change the index cache to use Distribution.Utils.Structured,
making Binary instances generically derived.

Almost as a feature generated 01-index.cache is never updated.
If you change the contents of the directory, you have to purge
01-index.cache file yourself.

[phadej]

I used the current state to build hadrian in "offline" mode, see https://gitlab.haskell.org/ghc/ghc/merge_requests/2348

I'll merge this soon to backport to 3.2 branch. Please review.

[typedrat]

This is exactly functionality that I was sorely missing the last time that I worked on trying to redesign/fix v2-install! Nice.

[phadej]

@typedrat could you expand on that? What breaks with packages: **/*.tar.gz? Could you open issue describing that problem. We might want to extend what can be specified in the project file still. There is overlap, and drawing a line is difficult.

Creating a temporary LocalRepo with a sdists of a cabal.project is simple and elegant solution, if we have a way to force cabal install versions from that LocalRepo, and not other configured repositories. The repository overriding / preference is not developed yet. #6139 is related. Yet we have to think about the problem better. I guess we

• often want package wide shadowing, i.e. "later" repository with pkg-verA removes all pkg from previous repositories
• but sometimes only per-version shadowing is enough, i.e. pkg-verA shadows only pkg-verB
• and with case of remote repositories we might want to selectively pick only some packages (or package-versions`) from overlays

I.e. we have develop a small configuration language for mconcat in

cabal/cabal-install/Distribution/Client/IndexUtils.hs

Lines 276 to 278 in 787b1f2

	let (pkgs, prefs) = mconcat pkgss
	prefs' = Map.fromListWith intersectVersionRanges
	[ (name, range) | Dependency name range _ <- prefs ]

i.e. parametrise PackageIndex merge

cabal/cabal-install/Distribution/Solver/Types/PackageIndex.hs

Lines 151 to 169 in 787b1f2

	-- | Merge two indexes.
	--
	-- Packages from the second mask packages of the same exact name
	-- (case-sensitively) from the first.
	--
	merge :: Package pkg => PackageIndex pkg -> PackageIndex pkg -> PackageIndex pkg
	merge i1@(PackageIndex m1) i2@(PackageIndex m2) =
	assert (invariant i1 && invariant i2) $
	mkPackageIndex (Map.unionWith mergeBuckets m1 m2)
	-- | Elements in the second list mask those in the first.
	mergeBuckets :: Package pkg => [pkg] -> [pkg] -> [pkg]
	mergeBuckets [] ys = ys
	mergeBuckets xs [] = xs
	mergeBuckets xs@(x:xs') ys@(y:ys') =
	case packageId x `compare` packageId y of
	GT -> y : mergeBuckets xs ys'
	EQ -> y : mergeBuckets xs' ys'
	LT -> x : mergeBuckets xs' ys

(EDIT: fixed the snippet)

[phadej]

Note: the overriding is not important for offline usage, as there you would specify the sole LocalRepo. Single repository is simple case :)

[hvr]

Looks good to me! Thanks for doing this!

[phadej]

Backported to 3.2 in #6461