Captive Stellar-Core wrapper server

[bartekn]

The current implementation of Captive Stellar-Core streams meta stream using a filesystem pipe. This implies that both Horizon and Stellar-Core have to be deployed to the same server. One of the disadvantages of such requirement is a need for detailed per-process monitoring to be able to connect potential issues (like memory leaks) to the specific service.

To solve this we can build a thin wrapper around CaptiveStellarCore backend that exposes a simple RPC/HTTP API similar to LedgerBackend interface:

go/exp/ingest/ledgerbackend/ledger_backend.go

Lines 26 to 40 in f77b3b5

	// LedgerBackend represents the interface to a ledger data store.
	type LedgerBackend interface {
	// GetLatestLedgerSequence returns the sequence of the latest ledger available
	// in the backend.
	GetLatestLedgerSequence() (sequence uint32, err error)
	// The first returned value is false when the ledger does not exist in a backend.
	GetLedger(sequence uint32) (bool, xdr.LedgerCloseMeta, error)
	// PrepareRange prepares the given range (including from and to) to be loaded.
	// Some backends (like captive stellar-core) need to initalize data to be
	// able to stream ledgers.
	PrepareRange(ledgerRange Range) error
	// IsPrepared returns true if a given ledgerRange is prepared.
	IsPrepared(ledgerRange Range) bool
	Close() error
	}

The full solution requires a server and a client that can consume the API.

A server can be a subcommand on horizon that exposes the CaptiveStellarCore object API using either the net/rpc package or HTTP. It has the same require params as CaptiveStellarCore. A server is a singleton, meaning it can start a single stellar-core process, thus should be used by a single Horizon instance.

A client, RemoteCaptiveStellarCore implementation of LedgerBackend interface is part of exp/ingest/ledgerbackend and accepts a connection params to a server.