Changes to support Edge Server #2202

snej · 2025-01-08T18:39:11Z

This branch contains all the LiteCore enhancements & fixes needed to implement Edge Server.

IMPORTANT: There are also changes in the EE repo: https://github.com/couchbase/couchbase-lite-core-EE/pull/47

Removed the REST server code, and a few support classes like RequestResponse; that's now in Edge Server.
Modified RESTSyncListener to avoid using RequestResponse, which simplified the code. (That's in the EE repo.)
Some API changes to C4Listener since it no longer supports serving the REST API.
The replicator now uses the new DatabasePool. By default, DBAccess will create its own pool of size 1, but an alternate Replicator constructor gives an existing shared pool for it to use. This will greatly reduce the overhead of running lots of replicators at once.
More stuff, described in the individual commits.

- Removed RESTListener - Misc. internal API additions / changes / fixes

WITH_ERROR can't use Catch's UNSCOPED_INFO(), but I've amended it to use WARN(), which will also show up in the test results.

If the pool is opened with a read-only db, then borrowWriteable() should immediately throw NotWriteable, not wait for a db to be returned and then throw Busy.

c4Replicator+Pool.hh defines factory functions to start a replicator given a DatabasePool* not a C4Database*. This shields callers from having to include c4DatabaseImpl.hh etc., which drag in a buttload of internal replicator headers.

Gives Task subclasses more control over when a finished task should be removed from the listing. Needed for ChangesTask.

Return value is true if index was created or replaced, false if nothing changed.

Requires updating EE repo

- clang-format - BorrowedCollection retains its C4Collection - Fixed warnings about out-of-order initializers - Fixed some params that needed C4NULLABLE - Removed unused SyncListener member variables

CBL_CORE_API shouldn't be used on the _definitions_ of data values, only their _declarations_.

Removing changes that became unnecessary adding some LIFETIMEBOUND

This occurs when EdgeServer sends an HTTP response but the client has closed the socket.

clang-format reformats the declaration of "challengeAuth" weirdly, with a two-line comment that has a '\' at the end of the first line. GCC warns about this with -Wall. Moved the doc-comment above the declaration to prevent this.

Internally, I streamlined the RecordEnumerator API by folding the other initialization parameters into the Options struct.

- ICU is now downloaded as part of the build - No more top level build server CMake wrapper is needed - LiteCore and LiteCoreStatic targets can be disabled to avoid having needless install output - Move sTempDir into LogEncoderTest.cc so it can be shared between CppTests and cbl-logtest

for EdgeServer, as part of creating Etags

For EdgeServer

It caused errors when LITECORE_BUILD_TESTS or LITECORE_BUILD_SHARED were turned off.

It's a no-op that got left in a commit (27d3530) by accident.

- HTTPLogic::parseHeaders() rejects control characters - Added constant for status 415 - Proper errors in TCPSocket::readHTTPBody()

snej · 2025-01-08T19:28:00Z

Jenkins builds are failing due to a CMake error:

[2025-01-08T18:54:13.724Z] CMake Error at cmake/platform_linux.cmake:75 (target_link_libraries):
[2025-01-08T18:54:13.724Z]   Target "LiteCoreObjects" links to:
[2025-01-08T18:54:13.724Z] 
[2025-01-08T18:54:13.724Z]     icu::icu4c
[2025-01-08T18:54:13.724Z] 
[2025-01-08T18:54:13.724Z]   but the target was not found.  Possible reasons include:

snej · 2025-01-08T21:02:17Z

Ubuntu in GitHub CI is failing a test due to a runtime exception:

DB Info SQLite message: statement aborts at 36: [CREATE UNIQUE INDEX IF NOT EXISTS "kv_.loopback.test_seqs" ON "kv_.loopback.test" (sequence)] attempt to write a readonly database

I know how to fix this...

The index is created lazily. If a collection doesn't have one, an operation using a read-only db from the pool might trigger creating it, which will fail because it's read-only. To work around this, DBAccess's constructor uses the writeable db to call call `getDocumentBySequence` on every collection, forcing the index to be created.

The search for ICU needs to happen on Linux desktop only. Also, ensure that the BUILD_ENTEPRISE flag is properly set now that we don't use the wrapper CMake project anymore (also on Windows this happened)

cbl-bot · 2025-01-09T00:58:03Z

Code Coverage Results:

Type	Percentage
branches	66.18
functions	78.11
instantiations	70.22
lines	77.36
regions	73.22

and bump compiler version to gcc 11 (required for using cbdeps ICU)

snej added 30 commits January 7, 2025 11:14

Replicator's DBAccess now uses a database pool

476f4d4

more db pool in replicator

9876bd4

c4ReplicatorImpl now supports DatabasePool

f1a453a

build fixes for CE and Xcode

209d43b

Changes for EdgeServer

ec21613

- Removed RESTListener - Misc. internal API additions / changes / fixes

SyncListener creates Replicator w/DatabasePool

b2e3b4b

SyncListener uses Task. Ensure Tasks get stopped.

096fd0d

DatabasePool logging

c9e665a

Support v3 protocol in P2P sync

3860ec9

CppTests pass again

caf75f2

Make WITH_ERROR info more visible in test output

515faee

WITH_ERROR can't use Catch's UNSCOPED_INFO(), but I've amended it to use WARN(), which will also show up in the test results.

DatabasePool improvements (Transaction)

aed2f47

Request/Response improvements

f9e992a

clang-format

5be017c

C4Listener: Per-database configuration

8fdee81

REST: Refactoring. Moved more code into EdgeServer.

8c99e38

DatabasePool: Proper error if pool is read-only

e20beea

If the pool is opened with a read-only db, then borrowWriteable() should immediately throw NotWriteable, not wait for a db to be returned and then throw Busy.

Internal API to start replicator with DatabasePool

bf00ffb

c4Replicator+Pool.hh defines factory functions to start a replicator given a DatabasePool* not a C4Database*. This shields callers from having to include c4DatabaseImpl.hh etc., which drag in a buttload of internal replicator headers.

Added HTTPListener::Task::listed()

ce9ffb7

Gives Task subclasses more control over when a finished task should be removed from the listing. Needed for ChangesTask.

Fixed uninitialized ptr in BorrowedCollection struct

4e2e837

Fixed misleading comment in c4ReplicatorTypes.h

833a63c

C4Collection::createIndex() now returns bool [API]

95b46fd

Return value is true if index was created or replaced, false if nothing changed.

Added missing include to HTTPListener.hh

56b3ffd

Added HTTPListener::findMatchingSyncProtocol()

2c08390

Requires updating EE repo

Fixed warnings, reformatted

0cac59d

- clang-format - BorrowedCollection retains its C4Collection - Fixed warnings about out-of-order initializers - Fixed some params that needed C4NULLABLE - Removed unused SyncListener member variables

Fixed nasty use-after-free bug in Headers move-assignment

ab5daf9

Fixing Xcode and Windows builds

4404356

Resolved GCC warnings w/CBL_CORE_API

98f4c0e

CBL_CORE_API shouldn't be used on the _definitions_ of data values, only their _declarations_.

cleanup

770663d

Removing changes that became unnecessary adding some LIFETIMEBOUND

Simplified implementation of Headers

3825962

snej and others added 14 commits January 7, 2025 11:14

TCPSocket: Don't log multiple identical errors

0ddbf87

This occurs when EdgeServer sends an HTTP response but the client has closed the socket.

C4Listener: Made stopping more robust

f5b4b10

HTTPListener: use task_id not pid in Task JSON description

8982590

C4DocEnumerator: Added startKey option [API]

4baf0a1

Internally, I streamlined the RecordEnumerator API by folding the other initialization parameters into the Options struct.

Attempt to fix MSVC link error w/ kC4Cpp_DefaultLog

bf15191

Fixed UBSan error calling C4Listener::shareDB()

2eff78e

Added C4Collection::getPurgeCount() [API]

1018db5

for EdgeServer, as part of creating Etags

HTTPListener: use "age_sece" not "started_on" in _active_tasks

0b7a3c2

For EdgeServer

Switch to official cbdeps build of icu

9ff9472

CMake: Fixed LITECORE_SANITIZE

bcf5f25

It caused errors when LITECORE_BUILD_TESTS or LITECORE_BUILD_SHARED were turned off.

Removed unnecessary/conflicting change in Replicator.cc

72c2ec8

It's a no-op that got left in a commit (27d3530) by accident.

Some HTTP fixes

628e64a

- HTTPLogic::parseHeaders() rejects control characters - Added constant for status 415 - Proper errors in TCPSocket::readHTTPBody()

snej requested review from jianminzhao, borrrden and callumbirks January 8, 2025 18:47

effin' clang-format again

d25c3c2

snej and others added 3 commits January 8, 2025 13:21

Updated fleece

a812511

Fix Android build issue

ba91d57

The search for ICU needs to happen on Linux desktop only. Also, ensure that the BUILD_ENTEPRISE flag is properly set now that we don't use the wrapper CMake project anymore (also on Windows this happened)

Add missing header in SyncListenerTest

972bde5

and bump compiler version to gcc 11 (required for using cbdeps ICU)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Changes to support Edge Server #2202

Changes to support Edge Server #2202

snej commented Jan 8, 2025 •

edited

Loading

snej commented Jan 8, 2025

snej commented Jan 8, 2025 •

edited

Loading

cbl-bot commented Jan 9, 2025

Changes to support Edge Server #2202

Are you sure you want to change the base?

Changes to support Edge Server #2202

Conversation

snej commented Jan 8, 2025 • edited Loading

snej commented Jan 8, 2025

snej commented Jan 8, 2025 • edited Loading

cbl-bot commented Jan 9, 2025

snej commented Jan 8, 2025 •

edited

Loading

snej commented Jan 8, 2025 •

edited

Loading