What is <YOUR_METLO_COLLECTOR_URL> ?

[ZeRego]

Hi there 👋

I setup Metlo locally using docker compose ( read me instructions ).

git clone https://github.com/metlo-labs/metlo.git
cd metlo
ENCRYPTION_KEY="some random string" EXPRESS_SECRET="some random string" docker-compose up -d

I'm also running my local app via docker compose and followed your intructions for a node connection.

var metlo = require("metlo")
metlo(<YOUR_METLO_API_KEY>, <YOUR_METLO_COLLECTOR_URL>)

First problem/question

Can't find in your docs what the YOUR_METLO_COLLECTOR_URL should be.
From looking at your docker compose file it looks like collector is also an alias for injestor and looking at the node library the value expected is just the host.
Not sure if I can just pass the host or also need to include the port for the injestor (8081).

Suggestion:

• update names to be consistent
• show the YOUR_METLO_COLLECTOR_URL value when the user is on the connections page or when creating a key

Second problem/question

Since I'm running both apps in docker I can't point to localhost. I tried using http://host.docker.internal but without success. Do you know what I should set as the YOUR_METLO_COLLECTOR_URL in my case?

[AHarmlessPyro]

HI @ZeRego
YOUR_METLO_COLLECTOR_URL refers a URL that is accessible to a metlo ingestor. Since you're using the docker compose method, by default it should be exposing the collector at http://localhost:8081. You can test if it's accessible by performing a GET request to http://localhost:8081/api/v1, and if it's working correctly, then it should respond with a 200 OK response.

curl --location --request GET 'http://localhost:8081/api/v1'

So, while running on docker for local applications, you should be able to use http://localhost:8081 as the collector url.

Thanks for your suggestions. We still have to make our docs better and surface information to users more intuitively 😅

[ZeRego]

Running that curl from my machine returns a 200.
Running that curl from the docker container, where my app runs, doesn't return 200

# curl --location --request GET 'http://localhost:8081/api/v1'
curl: (7) Failed to connect to localhost port 8081: Connection refused

# curl --location --request GET 'http://host.docker.internal:8081'
Client unauthorized

Do I need to change the network_mode on my app docker compose?

-- Update --

Looks like using the host.docker.internal worked, I didn't have the rest of the path before:

# curl --location --request GET 'http://host.docker.internal:8081/api/v1'
OK

So setting YOUR_METLO_COLLECTOR_URL to http://host.docker.internal:8081 should work. But I don't see any data showing up on the Metlo dashboards after using my app 🤔 is there a way to enable debug logs in your node library?

[AHarmlessPyro]

That's cool ! I was going to recommend making a docker bridge in case using host.docker.internal didn't work.

Okay. Right now there isn't a way to perform any debug logs. What you could try is running something like this :

curl --location --request POST '<YOUR_METLO_ADDRESS>/api/v1/log-request/single' \
--header 'Authorization: <YOUR_METLO_API_KEY>' \
--header 'Content-Type: application/json' \
--data-raw '{
    "request": {
        "url": {
            "host": "test-ecommerce.metlo.com",
            "path": "/cart",
            "parameters": []
        },
        "headers": [
            {
                "name": "X-API-KEY",
                "value": "e31f84ba-d92a-419b-b7ba-ef5e9df982f1"
            }
        ],
        "method": "POST",
        "body": ""
    },
    "response": {
        "status": 200,
        "headers": [
            {
                "name": "content-type",
                "value": "application/json; charset=utf-8"
            }
        ],
        "body": "{\"ok\": true, \"cart_uuid\": \"c423daed-9213-40ff-8fff-4691d840e290\", \"msg\": \"New Cart.\"}"
    },
    "meta": {
        "environment": "production",
        "incoming": true,
        "source": "17.99.145.104",
        "sourcePort": 17319,
        "destination": "76.47.25.189",
        "destinationPort": 443
    }
}'

This is a known good example, and should help us diagnose if the issue is in the credentials or metlo for node.

[ZeRego]

The issue here is that you require some way of collecting data from the container that you want to monitor.

We already have that part. You can check my commit and branch:
lightdash/lightdash@226ddd0

I double checked the env variables are correct and I get the log:
2022-11-10 16:05:47 [Lightdash] info: Started Metlo connection: http://host.docker.internal:8081

[ZeRego]

What you could try is running something like this :

Running that curl from my container worked and I can see the Metlo dashboard updated. That means the key and host are correct and the problem must be in my code or in the metlo library. 🤔

[AHarmlessPyro]

The issue here is that you require some way of collecting data from the container that you want to monitor.

We already have that part. You can check my commit and branch: lightdash/lightdash@226ddd0

I double checked the env variables are correct and I get the log: 2022-11-10 16:05:47 [Lightdash] info: Started Metlo connection: http://host.docker.internal:8081

My bad, I didn't notice that you were running on node. I have amended the response as such.

As for the second thing,

What you could try is running something like this :

Running that curl from my container worked and I can see the Metlo dashboard updated. That means the key and host are correct and the problem must be in my code or in the metlo library. 🤔

Looking at the commit you mentioned, can you try importing and initializing metlo before express ? I think that's the issue here. Metlo ingestor needs to intercept traces in express and to properly capture those, we need metlo to be imported initialized before express is imported.

[ZeRego]

Metlo ingestor needs to intercept traces in express and to properly capture those, we need metlo to be imported initialized before express is imported.

That doesn't seem code friendly😞
We have eslint rule import/first to have all the imports at the top and then the executable code. So that would require us to disable eslint for those lines or change our file structure to guarantee express is only imported afterwards.
It would be great if there was an express-metlo package that was initialized like other express plugins.

const app = express();
app.use(metlo);
...

After disabling the eslint I was able to get just 2 requests, out hundreds, logged in Metlo. And I can see that is not picking up the correct route paths.
First request shows the path '/health' when the full path is '/api/v1/health'
Second request shows the path / when the full path is `'/api/v1/user/'

We have our routes broken down across multiple files so I suppose you only pick the last part of the path.
Example:

// index.ts
const app = express();
app.use('/api/v1', apiV1Router);

// apiV1.ts
export const apiV1Router = express.Router();
apiV1Router.use('/user', userRouter);

// user.ts
export const userRouter = express.Router();
userRouter.get('/', (req, res) => {
 ...
});

[akshay288]

Hey @ZeRego sorry about that! We have a fix ready here: #101 . I'll let you know when we deploy the new version. Apologies again for the back and forth, that was a really dumb bug... We're fairly new so we're still ironing out the quirks.

Also, thanks for all the feedback! We'll incorporate it all in our docs and setup process in the next few days...

[akshay288]

Also I'd love to get on a call to get any other feedback you have :) Let me know if you'd be up for it!

[AHarmlessPyro]

Hey @ZeRego, We've updated our metlo package, and it should be able to capture all the routes. Let us know if there are any other issues you encounter. We'll do our best to fix them asap 😃

[AHarmlessPyro]

Hey @ZeRego, let us know if there's any other ways we can help you or improve metlo. In the meantime, I'll be closing this issue.