Fix ListProjects when ContentLength header is not defined #26

novln · 2023-09-15T09:23:25Z

Hello 👋,

Currently, it's not possible to use the ListProjects without receiving an "object not found" error (but not always).

This error is returned because https://console.neon.tech/api/v2/projects doesn't return the ContentLength when the list of projects is quite large.

From my understanding, it was a protection that you made for the first version of Neon API but looking at the issue, it seems this behavior was fixed ?!

Have a great day and thank you for taking the time to review this.

kislerdm · 2023-09-16T12:29:05Z

@novln Hey Thomas! Thank you for your contribution!

May I kindly ask to share an example/add a test case? Please include the content of the raw response (including headers) when you make an API call directly, without SDK.

It's important to understand if the issue shall be opened in the Neon repo. For context, AFAIK the Content-Length header is expected unless

the header Transfer-Encoding is specified,
or the response status is 204,
or the response status is 304 and response size differs compared to the which would have been returned had the status been 200.

For more info, please check the rfc.

Thanks!
Dmitry

novln · 2023-09-18T13:52:59Z

Hello 👋

May I kindly ask to share an example/add a test case?

Yeah, no worries. I've offuscated some stuff but it shouldn't tamper the example.

First test case

For the first test case, it's working fine with the SDK when we have few neon projects:

curl -vv --request GET --url 'https://console.neon.tech/api/v2/projects' --header 'accept: application/json' --header 'authorization: Bearer foobar'

*   Trying 3.16.200.69:443...
* Connected to console.neon.tech (3.16.200.69) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=console.neon.tech
*  start date: Aug 16 08:54:03 2023 GMT
*  expire date: Nov 14 08:54:02 2023 GMT
*  subjectAltName: host "console.neon.tech" matched cert's "console.neon.tech"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://console.neon.tech/api/v2/projects
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: console.neon.tech]
* [HTTP/2] [1] [:path: /api/v2/projects]
* [HTTP/2] [1] [user-agent: curl/8.3.0]
* [HTTP/2] [1] [accept: application/json]
* [HTTP/2] [1] [authorization: Bearer foobar]
> GET /api/v2/projects HTTP/2
> Host: console.neon.tech
> User-Agent: curl/8.3.0
> accept: application/json
> authorization: Bearer foobar
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/2 200
< date: Mon, 18 Sep 2023 13:06:22 GMT
< content-type: application/json
< content-length: 1451
< vary: Origin
< x-neon-ret-request-id: cebb09d8f969684070df21be2af5f1ac
< strict-transport-security: max-age=15724800; includeSubDomains
<
{"projects":[{"id":"round-moon-89447148","platform_id":"aws","region_id":"aws-eu-central-1","name":"koyeb-2d8b096d-04cf-477d-b833-da448596236b","provisioner":"k8s-pod","default_endpoint_settings":{"autoscaling_limit_min_cu":0.25,"autoscaling_limit_max_cu":0.25,"suspend_timeout_seconds":300},"pg_version":15,"proxy_host":"eu-central-1.aws.neon.tech","branch_logical_size_limit":204800,"branch_logical_size_limit_bytes":214748364800,"store_passwords":true,"active_time":0,"cpu_used_sec":0,"creation_source":"console","created_at":"2023-09-18T12:38:50Z","updated_at":"2023-09-18T13:04:05Z","synthetic_storage_size":32658984,"quota_reset_at":"2023-10-01T00:00:00Z","owner_id":"7c635b92-d0c1-47ad-98c4-f93f9f93bc77"},{"id":"late-dawn-81818886","platform_id":"aws","region_id":"aws-eu-central-1","name":"koyeb-09a0fa5a-9e7f-470c-9770-015fca5d0cd0","provisioner":"k8s-pod","default_endpoint_settings":{"autoscaling_limit_min_cu":0.25,"autoscaling_limit_max_cu":0.25,"suspend_timeout_seconds":300},"pg_version":15,"proxy_host":"eu-central-1.aws.neon.tech","branch_logical_size_limit":204800,"branch_logical_size_limit_bytes":214748364800,"store_passwords":true,"active_time":1236,"cpu_used_sec":309,"creation_source":"console","created_at":"2023-09-15T12:38:03Z","updated_at":"2023-09-18T07:13:57Z","synthetic_storage_size":32691728,"quota_reset_at":"2023-10-01T00:00:00Z","owner_id":"7c635b92-d0c1-47ad-98c4-f93f9f93bc77"}],"pagination":{"cursor":"late-dawn-81818886"}}

Second test case

For the second test case, it's not working with the SDK when we have a lot of neon projects (+75):

curl -vv --request GET --url 'https://console.neon.tech/api/v2/projects' --header 'accept: application/json' --header 'authorization: Bearer foobar'

*   Trying 3.16.200.69:443...
* Connected to console.neon.tech (3.16.200.69) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=console.neon.tech
*  start date: Aug 16 08:54:03 2023 GMT
*  expire date: Nov 14 08:54:02 2023 GMT
*  subjectAltName: host "console.neon.tech" matched cert's "console.neon.tech"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://console.neon.tech/api/v2/projects
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: console.neon.tech]
* [HTTP/2] [1] [:path: /api/v2/projects]
* [HTTP/2] [1] [user-agent: curl/8.3.0]
* [HTTP/2] [1] [accept: application/json]
* [HTTP/2] [1] [authorization: Bearer foobar]
> GET /api/v2/projects HTTP/2
> Host: console.neon.tech
> User-Agent: curl/8.3.0
> accept: application/json
> authorization: Bearer foobar
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/2 200
< date: Mon, 18 Sep 2023 13:12:59 GMT
< content-type: application/json
< vary: Origin
< x-neon-ret-request-id: 4aa14891fde5ee01b6e94880e159fced
< strict-transport-security: max-age=15724800; includeSubDomains
<
{"projects":[{"id":"nameless-mud-28966201","platform_id":"aws","region_id":"aws-eu-central-1","name":"koyeb-10e64236-5cbd-4d33-b07d-65402971e067","provisioner":"k8s-pod","default_endpoint_settings":{"autoscaling_limit_min_cu":0.25,"autoscaling_limit_max_cu":0.25,"suspend_timeout_seconds":300},"pg_version":15,"proxy_host":"eu-central-1.aws.neon.tech","branch_logical_size_limit":204800,"branch_logical_size_limit_bytes":214748364800,"store_passwords":true,"active_time":0,"cpu_used_sec":0,"creation_source":"console","created_at":"2023-09-18T13:11:57Z","updated_at":"2023-09-18T13:11:57Z","quota_reset_at":"2023-10-01T00:00:00Z","owner_id":"7c635b92-d0c1-47ad-98c4-f93f9f93bc77"},{"id":"broad-recipe-99988920","platform_id":"aws","region_id":"aws-eu-central-1","name":"koyeb-3bb3f597-7583-4b49-ae9d-3578be60fc8f","provisioner":"k8s-pod","default_endpoint_settings":{"autoscaling_limit_min_cu":0.25,"autoscaling_limit_max_cu":0.25,"suspend_timeout_seconds":300},"pg_version":15,"proxy_host":"eu-central-1.aws.neon.tech","branch_logical_size_limit":204800,"branch_logical_size_limit_bytes":214748364800,"store_passwords":true,"active_time":0,"cpu_used_sec":0,"creation_source":"console","created_at":"2023-09-18T13:11:57Z","updated_at":"2023-09-18T13:11:57Z","quota_reset_at":"2023-10-01T00:00:00Z","owner_id":"7c635b92-d0c1-47ad-98c4-f93f9f93bc77"},{"id":"round-queen-07796840","platform_id":"aws","region_id":"aws-eu-central-1","name":"koyeb-92cc8db7-2919-4e71-b8f4-5820c6f6e708","provisioner":"k8s-pod","default_endpoint_settings":{"autoscaling_limit_min_cu":0.25,"autoscaling_limit_max_cu":0.25,"suspend_timeout_seconds":300},"pg_version":15,"proxy_host":"eu-central-1.aws.neon.tech","branch_logical_size_limit":204800,"branch_logical_size_limit_bytes":214748364800,"store_passwords":true,"active_time":0,"cpu_used_sec":0,"creation_source":"console","created_at":"2023-09-18T13:11:56Z","updated_at":"2023-09-18T13:11:56Z","quota_reset_at":"2023-10-01T00:00:00Z","owner_id":"7c635b92-d0c1-47ad-98c4-f93f9f93bc77"},{"id":"round-night-89986993","platform_id":"aws","region_id":"aws-eu-central-1","name":"koyeb-7ff10233-0f02-4b5d-854b-5e4fe8710520","provisioner":"k8s-pod","default_endpoint_settings":{"autoscaling_limit_min_cu":0.25,"autoscaling_limit_max_cu":0.25,"suspend_timeout_seconds":300},"pg_version":15,"proxy_host":"eu-central-1.aws.neon.tech","branch_logical_size_limit":204800,"branch_logical_size_limit_bytes":214748364800,"store_passwords":true,"active_time":0,"cpu_used_sec":0,"creation_source":"console","created_at":"2023-09-18T13:11:56Z","updated_at":"2023-09-18T13:11:56Z","quota_reset_at":"2023-10-01T00:00:00Z","owner_id":"7c635b92-d0c1-47ad-98c4-f93f9f93bc77"},{"id":"long-frog-97538497","platform_id":"aws","region_id":"aws-eu-central-1","name":"koyeb-3685efcf-c068-4e06-ad79-b1360d0e8df9","provisioner":"k8s-pod","default_endpoint_settings":{"autoscaling_limit_min_cu":0.25,"autoscaling_limit_max_cu":0.25,"suspend_timeout_seconds":300},"pg_version":15,"proxy_host":"eu-central-1.aws.neon.tech","branch_logical_size_limit":204800,"branch_logical_size_limit_bytes":214748364800,"store_passwords":true,"active_time":0,"cpu_used_sec":0,"creation_source":"console","created_at":"2023-09-18T13:11:55Z","updated_at":"2023-09-18T13:11:55Z","quota_reset_at":"2023-10-01T00:00:00Z","owner_id":"7c635b92-d0c1-47ad-98c4-f93f9f93bc77"},{"id":"dawn-cherry-43934068","platform_id":"aws","region_id":"aws-eu-central-1","name":"koyeb-d52d52fb-1964-41b1-b1fd-94008aed4bfa","provisioner":"k8s-pod","default_endpoint_settings":{"autoscaling_limit_min_cu":0.25,"autoscaling_limit_max_cu":0.25,"suspend_timeout_seconds":300},"pg_version":15,"proxy_host":"eu-central-1.aws.neon.tech","branch_logical_size_limit":204800,"branch_logical_size_limit_bytes":214748364800,"store_passwords":true,"active_time":0,"cpu_used_sec":0,"creation_source":"console","created_at":"2023-09-18T13:11:54Z","updated_at":"2023-09-18T13:11:54Z","quota_reset_at":"2023-10-01T00:00:00Z","owner_id":"7c635b92-d0c1-47ad-98c4-f93f9f93bc77"},{"id":"damp-queen-20144320","platform_id":"aws","region_id":"aws-eu-central-1","name":"koyeb-8c3a6c21-d08b-4751-98a9-2fbd76836831","provisioner":"k8s-pod","default_endpoint_settings":{"autoscaling_limit_min_cu":0.25,"autoscaling_limit_max_cu":0.25,"suspend_timeout_seconds":300},"pg_version":15,"proxy_host":"eu-central-1.aws.neon.tech","branch_logical_size_limit":204800,"branch_logical_size_limit_bytes":214748364800,"store_passwords":true,"active_time":0,"cpu_used_sec":0,"creation_source":"console","created_at":"2023-09-18T13:11:54Z","updated_at":"2023-09-18T13:11:54Z","quota_reset_at":"2023-10-01T00:00:00Z","owner_id":"7c635b92-d0c1-47ad-98c4-f93f9f93bc77"},{"id":"restless-bush-93774803","platform_id":"aws","region_id":"aws-eu-central-1","name":"koyeb-466edecf-a240-4725-bb2e-07ea56ffda88","provisioner":"k8s-pod","default_endpoint_settings":{"autoscaling_limit_min_cu":0.25,"autoscaling_limit_max_cu":0.25,"suspend_timeout_seconds":300},"pg_version":15,"proxy_host":"eu-central-1.aws.neon.tech","branch_logical_size_limit":204800,"branch_logical_size_limit_bytes":214748364800,"store_passwords":true,"active_time":0,"cpu_used_sec":0,"creation_source":"console","created_at":"2023-09-18T13:11:53Z","updated_at":"2023-09-18T13:11:53Z","quota_reset_at":"2023-10-01T00:00:00Z","owner_id":"7c635b92-d0c1-47ad-98c4-f93f9f93bc77"},{"id":"cold-mountain-00347893","platform_id":"aws","region_id":"aws-eu-central-1","name":"koyeb-437c2831-d9dc-45d4-9a3b-2f9596b3a7d4","provisioner":"k8s-pod","default_endpoint_settings":{"autoscaling_limit_min_cu":0.25,"autoscaling_limit_max_cu":0.25,"suspend_timeout_seconds":300},"pg_version":15,"proxy_host":"eu-central-1.aws.neon.tech","branch_logical_size_limit":204800,"branch_logical_size_limit_bytes":214748364800,"store_passwords":true,"active_time":0,"cpu_used_sec":0,"creation_source":"console","created_at":"2023-09-18T13:11:53Z","updated_at":"2023-09-18T13:11:53Z","quota_reset_at":"2023-10-01T00:00:00Z","owner_id":"7c635b92-d0c1-47ad-98c4-f93f9f93bc77"},{"id":"shy-cell-03354258","platform_id":"aws","region_id":"aws-eu-central-1","name":"koyeb-608f9d62-2d6f-4d22-bd51-8e6e31338757","provisioner":"k8s-pod","default_endpoint_settings":{"autoscaling_limit_min_cu":0.25,"autoscaling_limit_max_cu":0.25,"suspend_timeout_seconds":300},"pg_version":15,"proxy_host":"eu-central-1.aws.neon.tech","branch_logical_size_limit":204800,"branch_logical_size_limit_bytes":214748364800,"store_passwords":true,"active_time":0,"cpu_used_sec":0,"creation_source":"console","created_at":"2023-09-18T13:11:52Z","updated_at":"2023-09-18T13:11:52Z","quota_reset_at":"2023-10-01T00:00:00Z","owner_id":"7c635b92-d0c1-47ad-98c4-f93f9f93bc77"}],"pagination":{"cursor":"shy-cell-03354258"}}

As you can see, content-length is not defined in the response, which returns the "object not found" error.

Content-Length header

It's important to understand if the issue shall be opened in the Neon repo [...] For context, AFAIK the Content-Length header is expected [...]

I kindly disagree because, from my understanding, the Content-Length header is now optional as written in RFC 7540 and RFC 9113 for HTTP/2 (which your SDK users could use).

Nonetheless, if it was mandatory and Neon should return this header, I still think that we should remove this workaround because, from my understanding, the root issue is fixed. Or at least, make it optional/ignored for listing projects.

I would very much like to not maintain a fork of this SDK. I hope we'll find some middle ground on this.

Have a nice day,
Thomas.

kislerdm · 2023-09-18T21:11:27Z

@novln Hey Thomas! Thank you very much for your contribution and your examples!

I confirm the issue: I experienced it when tried to list my projects for acceptance tests of the Neon terraform provider. Unexpected behavior occurs when the number of projects exceeds two.

I would very much like to not maintain a fork of this SDK. I hope we'll find some middle ground on this.

No worries, I reckon we found it already - your proposal will be released with new SDK version tomorrow. Your contribution will not only unblock you, but it will also enable release of new version of the terraform provider to unblock a couple of teams relying on it! Thanks a lot for your help!

Speaking of maintenance, would you be open to collaborate and maintain the SDK together? I believe that it'd greatly benefit its quality and reliability. WDYT?

Regards,
Dmitry

novln · 2023-09-19T12:38:57Z

Glad to hear it 😃

Should I update Test_client_requestHandler/unhappy_path in tests to fix the broken test case? Or do you prefer to update it when you'll release a new version?

Speaking of maintenance, would you be open to collaborate and maintain the SDK together?

Absolutely, but I should warn you that I'll only collaborate through Koyeb, my current employer.
I'm not using this SDK on a personal matter.

kislerdm · 2023-09-19T17:15:47Z

@novln Hey Thomas!

Should I update Test_client_requestHandler/unhappy_path in tests to fix the broken test case? Or do you prefer to update it when you'll release a new version?

The SDK code is generated using the "generator" app. Hence, the delivery of proposed changes will require the following steps:

The following LoC are deleted:

The SDK codebase is regenerated:

make generate-sdk

Absolutely, but I should warn you that I'll only collaborate through Koyeb, my current employer. I'm not using this SDK on a personal matter.

It's totally fine as long as your employer is OK with the SDK distributed under the MIT licence. BTW is your employer hiring? :) I'm looking for new gig, and Koyeb's product seems promising, and the company values are quite in line with mine. WDYT, would you be open to introducing me to the founders to discuss if my profile could be a fit? Thanks!

…ader. Closes #26 Signed-off-by: Dmitry Kisler <[email protected]>

fix: remove protection against non-existing object on API v2

7c1bca5

kislerdm added a commit that referenced this pull request Sep 19, 2023

fix: removes the GET '404' condition based on the 'Content-Length' he…

8824f60

…ader. Closes #26 Signed-off-by: Dmitry Kisler <[email protected]>

kislerdm mentioned this pull request Sep 19, 2023

Removes workaround for "200 response status" on "no object found" #27

Merged

kislerdm closed this in 77f268c Sep 19, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix ListProjects when ContentLength header is not defined #26

Fix ListProjects when ContentLength header is not defined #26

novln commented Sep 15, 2023 •

edited

Loading

kislerdm commented Sep 16, 2023 •

edited

Loading

novln commented Sep 18, 2023

kislerdm commented Sep 18, 2023 •

edited

Loading

novln commented Sep 19, 2023

kislerdm commented Sep 19, 2023 •

edited

Loading

Fix ListProjects when ContentLength header is not defined #26

Fix ListProjects when ContentLength header is not defined #26

Conversation

novln commented Sep 15, 2023 • edited Loading

kislerdm commented Sep 16, 2023 • edited Loading

novln commented Sep 18, 2023

First test case

Second test case

Content-Length header

kislerdm commented Sep 18, 2023 • edited Loading

novln commented Sep 19, 2023

kislerdm commented Sep 19, 2023 • edited Loading

novln commented Sep 15, 2023 •

edited

Loading

kislerdm commented Sep 16, 2023 •

edited

Loading

kislerdm commented Sep 18, 2023 •

edited

Loading

kislerdm commented Sep 19, 2023 •

edited

Loading