HI Team,

We have configured metlo in GCP and daemonset in GKE. Data is not getting exported to application. KIndly help us on this. Here are the attached log of one of pod

21/10/2022 -- 06:56:59 - - [ERRCODE: SC_ERR_STATS_LOG_GENERIC(278)] - eve.stats: stats are disabled globally: set stats.enabled to true. See https://suricata.readthedocs.io/en/suricata-6.0.5/configuration/suricata-yaml.html#stats 21/10/2022 -- 06:56:59 - - Running in live mode, activating unix socket 21/10/2022 -- 06:56:59 - - 1 rule files processed. 1 rules successfully loaded, 0 rules failed 21/10/2022 -- 06:56:59 - - Threshold config parsed: 0 rule(s) found 21/10/2022 -- 06:56:59 - - 1 signatures processed. 0 are IP-only rules, 0 are inspecting packet payload, 1 inspect application layer, 0 are decoder event only 21/10/2022 -- 06:56:59 - - Going to use 1 thread(s) 21/10/2022 -- 06:56:59 - - Running in live mode, activating unix socket 21/10/2022 -- 06:56:59 - - Using unix socket file '/var/run/suricata/suricata-command.socket' 21/10/2022 -- 06:56:59 - - all 1 packet processing threads, 2 management threads initialized, engine started. 21/10/2022 -- 06:56:59 - - All AFP capture threads are running. 21/10/2022 -- 06:56:58 - - This is Suricata version 6.0.5 RELEASE running in SYSTEM mode 21/10/2022 -- 06:56:58 - - CPUs/cores online: 1 21/10/2022 -- 06:56:59 - - Found an MTU of 1460 for 'eth0' 21/10/2022 -- 06:56:59 - - Found an MTU of 1460 for 'eth0' 21/10/2022 -- 06:56:59 - - Setting logging socket of non-blocking in live mode. 21/10/2022 -- 06:56:59 - - eve-log output device (unix_stream) initialized: /etc/suricata-logs/eve.sock 21/10/2022 -- 06:56:59 - - JsonRdpLog logger not enabled: protocol rdp is disabled 21/10/2022 -- 06:56:59 - - JsonIKEv2Log logger not enabled: protocol ikev2 is disabled 21/10/2022 -- 06:56:59 - - JsonKRB5Log logger not enabled: protocol krb5 is disabled 21/10/2022 -- 06:56:59 - - JsonSNMPLog logger not enabled: protocol snmp is disabled 21/10/2022 -- 06:56:59 - - JsonRFBLog logger not enabled: protocol rfb is disabled 21/10/2022 -- 06:56:59 - - [ERRCODE: SC_ERR_STATS_LOG_GENERIC(278)] - eve.stats: stats are disabled globally: set stats.enabled to true. See https://suricata.readthedocs.io/en/suricata-6.0.5/configuration/suricata-yaml.html#stats

============================================================================================

metlo.yaml:

apiVersion: apps/v1 kind: DaemonSet metadata: name: metlo-app spec: selector: matchLabels: name: metlo-app template: metadata: labels: name: metlo-app spec: hostNetwork: true tolerations: # this toleration is to have the daemonset runnable on master nodes # remove it if your masters can't run pods - key: node-role.kubernetes.io/master effect: NoSchedule containers: - name: suricata-daemon image: metlo/suricata-daemon imagePullPolicy: Always securityContext: privileged: true env: - name: METLO_ADDR value: ------------ - name: METLO_KEY value: *****

Hi there 👋

I setup Metlo locally using docker compose ( read me instructions ).

git clone https://github.com/metlo-labs/metlo.git
cd metlo
ENCRYPTION_KEY="some random string" EXPRESS_SECRET="some random string" docker-compose up -d

I'm also running my local app via docker compose and followed your intructions for a node connection.

var metlo = require("metlo")
metlo(<YOUR_METLO_API_KEY>, <YOUR_METLO_COLLECTOR_URL>)

First problem/question

Can't find in your docs what the YOUR_METLO_COLLECTOR_URL should be. From looking at your docker compose file it looks like collector is also an alias for injestor and looking at the node library the value expected is just the host. Not sure if I can just pass the host or also need to include the port for the injestor (8081).

Suggestion:

• update names to be consistent
• show the YOUR_METLO_COLLECTOR_URL value when the user is on the connections page or when creating a key

Second problem/question

Since I'm running both apps in docker I can't point to localhost. I tried using http://host.docker.internal but without success. Do you know what I should set as the YOUR_METLO_COLLECTOR_URL in my case?

I deployed with Docker step by step in a cloud server, following the instructions:

Devtools network tab show this:

1. /auth/login (200 OK - POST) -> JWT
2. /_next/data/XXXXXXXXXXXX/index.json (307 Temporary Redirect - GET) -> no response
3. /_next/data/XXXXXXXXXXXX/login.json (200 OK - GET) ->

{
    "pageProps": {
        "localAuthAllowed": true
    },
    "__N_SSP": true
}

4. Nothing happens

I install metlo from npm

npm i -g @metlo/cli

Server: Ubuntu 22.04.1 LTS Nodejs: v12.22.9 Npm: 8.5.1 metlo: metlo/[email protected]

Error log

metlo --help
/usr/local/lib/node_modules/@metlo/cli/node_modules/@aws-sdk/client-ec2/dist-cjs/protocols/Aws_ec2.js:22372
        if (input.TagSpecifications?.length === 0) {
                                    ^

SyntaxError: Unexpected token '.'
    at wrapSafe (internal/modules/cjs/loader.js:915:16)
    at Module._compile (internal/modules/cjs/loader.js:963:27)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
    at Module.load (internal/modules/cjs/loader.js:863:32)
    at Function.Module._load (internal/modules/cjs/loader.js:708:14)
    at Module.require (internal/modules/cjs/loader.js:887:19)
    at require (internal/modules/cjs/helpers.js:74:18)
    at Object.<anonymous> (/usr/local/lib/node_modules/@metlo/cli/node_modules/@aws-sdk/client-ec2/dist-cjs/commands/AcceptAddressTransferCommand.js:8:19)
    at Module._compile (internal/modules/cjs/loader.js:999:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)

/opt/venv/lib/python3.10/site-packages/django/test/client.py:751: in post
    response = super().post(path, data=data, content_type=content_type, secure=secure, **extra)
/opt/venv/lib/python3.10/site-packages/django/test/client.py:407: in post
    return self.generic('POST', path, post_data, content_type,
/opt/venv/lib/python3.10/site-packages/django/test/client.py:473: in generic
    return self.request(**r)
/opt/venv/lib/python3.10/site-packages/django/test/client.py:719: in request
    self.check_exception(response)
/opt/venv/lib/python3.10/site-packages/django/test/client.py:580: in check_exception
    raise exc_value
/opt/venv/lib/python3.10/site-packages/django/core/handlers/exception.py:47: in inner
    response = get_response(request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <metlo.django.MetloDjango object at 0xffff911058a0>
request = <WSGIRequest: POST '/admin/module/group/group_name/delete/'>

    def __call__(self, request):
        response = self.get_response(request)
        params = request.GET if request.method == "GET" else request.POST
        dest_ip = request.META.get("SERVER_NAME") if \
            "1.0.0.127.in-addr.arpa" not in request.META.get("SERVER_NAME") else "localhost"
        src_ip = request.META.get("REMOTE_ADDR") if \
            "1.0.0.127.in-addr.arpa" not in request.META.get("REMOTE_ADDR") else "localhost"
>       source_port = request.environ["wsgi.input"].stream.raw._sock.getpeername()[1]
E       AttributeError: 'FakePayload' object has no attribute 'stream'

/opt/venv/lib/python3.10/site-packages/metlo/django.py:45: AttributeError

I've deployed docker version of Metlo and configured our Django application with the metlo module. After running our test suite I ended up with most of the tests failing with same AttributeError.

I'm trying to use metlo on my local host. I've followed the steps from README. After containers are started when I try to visit the web app, it returns a 500 error.

Do I need to set environment variables? can the team add those instructions to the README?

$ docker-compose up -d
WARNING: The BACKEND_URL variable is not set. Defaulting to a blank string.
WARNING: The SANDBOX_MODE variable is not set. Defaulting to a blank string.
WARNING: The DISABLE_LOGGING_STATS variable is not set. Defaulting to a blank string.
WARNING: The NUM_WORKERS variable is not set. Defaulting to a blank string.

-- snip --

We are seeing below error when we try to install metlo daemonset on K8S cluster. I can see similar issue on https://github.com/metlo-labs/metlo/issues/52#issuecomment-1296806302 Can you share solution.

STARTING starting suricata starting metlo 10/11/2022 -- 19:26:26 - - This is Suricata version 6.0.1 RELEASE running in SYSTEM mode 10/11/2022 -- 19:26:26 - - [ERRCODE: SC_ERR_SYSCALL(50)] - Failure when trying to get MTU via ioctl for 'eth0': No such device (19) 10/11/2022 -- 19:26:26 - - [ERRCODE: SC_ERR_SYSCALL(50)] - Failure when trying to get MTU via ioctl for 'eth0': No such device (19) 10/11/2022 -- 19:26:26 - - [ERRCODE: SC_ERR_SOCKET(200)] - Error connecting to socket "/tmp/eve.sock": Connection refused (will keep trying) 10/11/2022 -- 19:26:26 - - JsonRdpLog logger not enabled: protocol rdp is disabled 10/11/2022 -- 19:26:26 - - JsonIKEv2Log logger not enabled: protocol ikev2 is disabled 10/11/2022 -- 19:26:26 - - JsonKRB5Log logger not enabled: protocol krb5 is disabled 10/11/2022 -- 19:26:26 - - JsonSNMPLog logger not enabled: protocol snmp is disabled 10/11/2022 -- 19:26:26 - - JsonRFBLog logger not enabled: protocol rfb is disabled 10/11/2022 -- 19:26:26 - - [ERRCODE: SC_ERR_STATS_LOG_GENERIC(278)] - eve.stats: stats are disabled globally: set stats.enabled to true. See https://suricata.readthedocs.io/en/suricata-6.0.1/configuration/suricata-yaml.html#stats 10/11/2022 -- 19:26:27 - - [ERRCODE: SC_ERR_AFP_CREATE(190)] - Unable to find type for iface "eth0": No such device 10/11/2022 -- 19:26:27 - - all 2 packet processing threads, 2 management threads initialized, engine started. 10/11/2022 -- 19:26:27 - - [ERRCODE: SC_ERR_AFP_CREATE(190)] - Unable to find iface eth0: No such device 10/11/2022 -- 19:26:27 - - [ERRCODE: SC_ERR_AFP_CREATE(190)] - Couldn't init AF_PACKET socket, fatal error 10/11/2022 -- 19:26:27 - - [ERRCODE: SC_ERR_FATAL(171)] - thread W#01-eth0 failed

Greetings Metlo Team,

Is it possible to store PII information in obfuscated form? This will ensure user privacy.

for example:

{
"other_params":"...",
"email":"[email protected]",
"phone":"1234567890"
}

Obfuscated Form:

{
"other_params":"...",
"email":"[email protected]",
"phone":"xx3456xxxx"
}

Hi! With the nodejs agent when the POST request is sent to the metlo backend from nodejs agent the endpoint /api/v1/log-request/single is missing, so the current workaround is to add /api/v1/log-request/single to the host URL

Hello! I tried to start Metlo in local environment with docker-compose-local.yaml file: sudo docker-compose -f docker-compose-local.yaml up in repo root directory.

Firstly, I get the next error: Service 'ingestor' failed to buid: ADD failed: forbidden path outside the build context: ../../../backend (). So, I changed ADD ../../../backend backend to ADD ./backend backend, and ADD ../../../common common to ADD ./common common in all dockerfiles in deploy directory. And it works.

But the next error has been appeared, like in previous issue (https://github.com/metlo-labs/metlo/issues/32):

Could you please help me? Thanks!

I tried it on my Android repo but, almost all of the issued cases were false-positive. For example https://github.com/Drjacky/MVIModularizationTemplate/issues/19

It says there is an issue with kotlin-stdlib-1.4.31.jar but, neither my repo nor the internal library(root dependency) use that version:

Third-library: https://github.com/detekt/sarif4k/blob/main/build.gradle.kts

TODO:

• Cross product on multiple payloads in a single request
• UI for test generation
• More payload types
  • Also maybe split payloads into more manageable chunks/representative types

How does Metlo detect params?

Some request paths are marked with incorrect parameters.

The current path in Metlo: /api/{param1}/org/{param2} Expected /api/{param1}/org/onboardingStatus

Route source: https://github.com/lightdash/lightdash/blob/e6057c27c37659235e1617527e8a6e27881ee9bc/packages/backend/src/routers/organizationRouter.ts#L176

Could it use the express routes to detect the params instead of using some magic logic? It can even detect what is the name of the param instead of using generic param1 param2 etc eg: /user/:userUuid -> /user/{userUuid}