MATE: Manual

This manual will give you a detailed description of how MATE works.

/!\ As MATE's grammar has been changed the examples in this manual use the grammar of the upcoming version of mate (post 0.10.11). If you want to see the current version of mate select info and then view a version before May 2005.

Table of contents

Contents

  1. MATE: Manual
    1. Table of contents
    2. Introduction
    3. Attribute Value Pairs
    4. AVP lists
    5. MATE Analysis
      1. Mate's PDUs
        1. PDU data extraction
        2. Conditions on which to create PDUs
        3. Transforming the attributes of a PDU
        4. MATE's PDU tree
      2. Grouping Pdus together (Gop)
        1. What can belong to a Gop
        2. Start of a Gop
        3. What goes into the Gop's AVPL
        4. End of a Gop
        5. Gop's tree
      3. Grouping Gops together (Gog)
        1. Declaring a Group Of Groups
        2. Telling MATE what could be a Gog member
        3. Getting interesting data into the Gop
        4. Gog's tree
      4. AVPL Transformations
        1. Syntax
        2. Usage
        3. Operation
        4. MATE's manipulation tool
    6. About MATE

Introduction

MATE creates a filterable tree based on information contained in frames that share some relationship with information obtained from other frames. The way this relationships are made is described in a configuration file. The configuration file tells MATE what makes a PDU and how to relate it to other PDUs.

MATE analyzes each frame to extract relevant information from the "protocol" tree of that frame. The extracted information is contained in MATE PDUs; these contain a list of relevant attributes taken from the tree. From now on, I will use the term "PDU" to refer to the objects created by MATE containing the relevant information extracted from the frame; I'll use "frame" to refer to the "raw" information extracted by the various dissectors that pre-analyzed the frame.

For every PDU, MATE checks if it belongs to an existing "Group of PDUs" (Gop). If it does, it assigns the PDU to that Gop and moves any new relevant atributes to the Gop's attribute list. How and when do PDUs belong to Gops is described in the configuration file as well.

Every time a Gop is assigned a new PDU, MATE will check if it matches the conditions to make it belong to a "Group of Groups" (Gog). Naturally the conditions that make a Gop belong to a Gog are taken from the configuration file as well.

Once MATE is done analyzing the frame it will be able to create a "protocol" tree for each frame based on the PDUs, the Gops they belong to and naturally any Gogs the former belongs to.

How to tell MATE what to extract, how to group it and then how to relate those groups is made using AVPs and AVPLs.

Information in MATE is contained in Attribute/Value Pairs (AVPs). AVPs are made of two strings: the name and the value. AVPs are used in the configuration and there they have an operator as well. There are various ways AVPs can be matched against each other using those operators.

AVPs are grouped into AVP Lists (AVPLs). PDUs, GOPs and Gogs have an AVPL each. Their AVPLs will be matched in various ways against others coming from the configuration file.

MATE will be instructed how to extract AVPs from frames in order to create a PDU with an AVPL. It will be instructed as well, how to match that AVPL against the AVPLs of other similar PDUs in order to relate them. In MATE the relationship between PDUs is a Gop, it has an AVPL as well. MATE will be configured with other AVPLs to operate against the Gop's AVPL to relate Gops together into Gogs.

A good understanding on how AVPs and AVPLs work is fundamental to understand how MATE works.

Attribute Value Pairs

Information used by MATE to relate different frames is contained in Attribute/Value Pairs (AVPs). AVPs are made of two strings the name and the value. When AVPs are used in the configuration an operator is defined as well. There are various ways AVPs can be matched against each other using those operators. 

  avp_name="avp's value"
  another_name= "1234 is the value"

The name is a string used to refer to a "kind" of AVPs. Two AVPs won't match unless their names are identical.

You should not use uppercase characters in names or names that start with '.' or '_'. Capitalized names are reserved for configuration parameters (we'll call them keywords); nothing forbids you from using capitalized strings for other things as well but it probably would be confusing. I'll avoid using capitalized words for anything but the keywords in this document, the reference manual, the examples and the base library. Names that start with a '.' would be very confusing as well because AVPL transformations use names starting with a '.' to indicate they belong to the replacement AVPL.

The value is a string that is either set in the configuration (for configuration AVPs) or by wireshark while extracting interesting fields from a frame's tree. The values extracted from fields use the same representation as they do in filter strings except that no quotes are used.

The name can contain only alphanumeric characters, "_", and ".". The name ends with an operator.

The value will be dealt with as a string even if it is a number. If there are any spaces in the value, the value must be between quotes "". 

   ip_addr=10.10.10.11,
   tcp_port=1234,
   binary_data=01:23:45:67:89:ab:cd:ef,
   parameter12=0x23aa,
   parameter_with_spaces="this value has spaces"

The way two AVPs with the same name might match is described by the operator. Remember two AVPs won't match unless their names are identical. In MATE, match operations are always made between the extracted information and the configuration's AVPs.

Currently defined MATE's AVP match operators are:

AVP lists

An AVPL is a set of diverse AVPs that can be matched against other AVPLs. Every Pdu, Gop and Gog has an AVPL that contains the information regarding it. The rules that MATE uses to group Pdus and Gops are AVPL operations.

There will never be two identical AVPs in a given AVPL. But, we can have more than one AVP with the same name in an AVPL as long as their values are different.

Some AVPL examples: 

  ( addr=10.20.30.40, addr=192.168.0.1, tcp_port=21, tcp_port=32534, user_cmd=PORT, data_port=12344, data_addr=192.168.0.1)
  ( addr=10.20.30.40, addr=192.168.0.1, channel_id=22:23, message_type=Setup, calling_number=1244556673)
  ( addr=10.20.30.40, addr=192.168.0.1, ses_id=01:23:45:67:89:ab:cd:ef)
  ( user_id=pippo, calling_number=1244556673, assigned_ip=10.23.22.123)

In MATE there are two types of AVPLs:

Data AVPLs can be operated against operation AVPLs in various ways:

MATE Analysis

MATE's analisys of a frame is performed in three phases:

mate_analysis.png

The extraction and matching logic comes from MATE's configuration; MATE's configuration file is declared by the mate.config preference. By default it is an empty string which means: do not configure MATE.

The config file tells MATE what to look for in frames; How to make PDUs out of it; How do PDUs will be related to other similar PDUs into Gop; And how Gops relate into Gogs.

The MATE configuration file is a list of declarations there are 4 types of declarations: Transform, Pdu, Gop and Gog.

Mate's PDUs

MATE will look in the tree of every frame to see if there is useful data to extract, and if there is, it will create one or more PDU objects containing the useful information.

The first part of MATE's analysis is the "PDU extraction"; there are various "Action AVPLs" that are used to instruct MATE what has to be extracted from the current frame's tree into MATE's PDUs.

PDU data extraction

MATE will make a Pdu for each different proto field of Proto type present in the frame. MATE will fetch from the field's tree those fields that are defined in the Mate/Reference#Pdu declaration whose initial offset in the frame is within the boundaries of the current Proto and those of the given Transport and Payload statements. 

Pdu dns_pdu Proto dns Transport ip {
    Extract addr From ip.addr;
    Extract dns_id From dns.id;
    Extract dns_resp From dns.flags.response;
};

MATE will make a Pdu for each different proto field of Proto type present in the frame. MATE will fetch from the field's tree those fields that are defined in the Pdu AVPL whose initial offset in the frame is within the boundaries of the current Proto and those of the various assigned Transports.

dns_pane.png

Once MATE has found a Proto field for which to create a Pdu from the frame it will move backwards in the frame looking for the respective Transport fields. After that it will create AVPs named as each of those given in the rest of the AVPL for every instance of the fields declared as its values.

dns_pdu.png

Sometimes we need information from more than one Transport protocol. In that case MATE will check the frame looking backwards to look for the various Transport protocols in the given stack. MATE will choose only the closest transport boundary per "protocol" in the frame.

This way we'll have all Pdus for every Proto that appears in a frame match its relative transports. 

Pdu isup_pdu Proto isup Transport mtp3/ip {
        Extract m3pc From mtp3.dpc;
        Extract m3pc From mtp3.opc;
        Extract cic From isup.cic;
        Extract addr From ip.addr;
        Extract isup_msg From isup.message_type;
};

isup_over_mtp3_over_ip.png

This allows to assign the right Transport to the Pdu avoiding duplicate transport protocol entries (in case of tunneled ip over ip for example). 

Pdu ftp_pdu Proto ftp Transport tcp/ip {
        Extract addr From ip.addr;
        Extract port From tcp.port;
        Extract ftp_cmd From ftp.command;
};

ftp_over_gre.png

Other than the mandatory Transport there is also an optional Payload statement, which works pretty much as Transport but refers to elements after the Proto's range. It is useful in those cases where the payload protocol might not appear in a Pdu but nevertheless the Pdu belongs to the same category. 

Pdu mmse_over_http_pdu Proto http Transport tcp/ip {

        Payload mmse;

        Extract addr From ip.addr;
        Extract port From tcp.port;
        Extract method From http.request.method;
        Extract content From http.content_type;
        Extract http_rq From http.request;
        Extract resp From http.response.code;
        Extract host From http.host;
        Extract trx From mmse.transaction_id;
        Extract msg_type From mmse.message_type;
        Extract notify_status From mmse.status;
        Extract send_status From mmse.response_status;
};

mmse_over_http.png

Conditions on which to create PDUs

There might be cases in which we won't want MATE to create a PDU unless some of its extracted attributes meet or do not meet some criteria. For that we use the Criteria statements of the Pdu declarations. 

Pdu isup_pdu Proto isup Transport mtp3/ip {
    ...

   // MATE will create isup_pdu PDUs only when there is not a point code '1234'
   Criteria Reject Strict (m3pc=1234);
};

Pdu ftp_pdu Proto ftp Transport tcp/ip {
    ...

    // MATE will create ftp_pdu PDUs only when they go to port 21 of our ftp_server
    Criteria Accept Strict (addr=10.10.10.10, port=21);
};

The Criteria statement is given an action (Accept or Reject), a match mode (Strict, Loose or Every) and an AVPL against which to match the currently extracted one.

Transforming the attributes of a PDU

Once the fields have been extracted into the Pdu's AVPL, MATE will apply every declared transformation to it. The way transforms are applied and how they work is described later on. However it's useful to know that once the AVPL for the Pdu is created it might be transformed before being analyzed. That way we can massage the data to simplify the analysis.

MATE's PDU tree

Every successfully created Pdu will create a MATE tree. If the Pdu is unassigned the tree for the Pdu will contain just the Pdu's info, if it is assigned to a Gop the tree will also contain the Gop and eventually the Gog tree. 

mate dns_pdu:1
    dns_pdu: 1
        dns_pdu time: 3.750000
        dns_pdu Attributes
            dns_resp: 0
            dns_id: 36012
            addr: 10.194.4.11
            addr: 10.194.24.35

The Pdu's tree contains some filterable fileds

Grouping Pdus together (Gop)

Once MATE has created the Pdus it passes to the Pdu analysis phase. During the PDU analysis phase MATE will try to group Pdus of the same type into Groups of Pdus (aka. Gops) and copy from the Pdu's AVPL to the Gop's AVPL some of its AVPs.

pdu_analysis.png

What can belong to a Gop

Given a Pdu the first thing MATE will do is to check if there is any Action=GopDef in the configuration for the given Pdu type. If so it will use it to match it against the Pdu's AVPL; if they don't match the analysis phase is done. If there is a match the match is the Gop's candidate key which will be used to search the Gop's index for the Gop to which to assign the current PDU. If there's is no such Gop and this Pdu does not match the GopStart criteria the Pdu will remain unassigned and the analysis phase will be done. 

Gop ftp_ses On ftp_pdu Match (addr, addr, port, port);
Gop dns_req On dns_pdu Match (addr, addr, dns_id);
Gop isup_leg On isup_pdu Match (m3pc, m3pc, cic);

Start of a Gop

If there was a match the candidate key will be used to search the Gop's index to see if there is already a session matching the Gop's key the same way. If there is such a match in the Gops collection, and the PDU doesn't match the GopStart AVPL for its kind, the PDU will be assigned to the matching Gop. If it is a GopStart match it will check whether or not the Gop has being already stopped. If the Gop has being stopped a new Gop will be created and will replace the old one in the Gop's index. 

Gop ftp_ses On ftp_pdu Match (addr, addr, port, port) {
    Start (ftp_cmd=USER);
};

Gop dns_req On dns_pdu Match (addr, addr, dns_id) {
    Start (dns_resp=0);
};

Gop isup_leg On isup_pdu Match (m3pc, m3pc, cic) {
    Start (isup_msg=1);
};

If no Start is given for a Gop, a Pdu whose AVPL matches an existing Gog's key will act as the start of a Gop.

What goes into the Gop's AVPL

Once we know a Gop exists and the Pdu has been assigned to it, MATE will copy into the Gop's AVPL all the attributes matching the key plus any AVPs of the Pdu's AVPL matching the Extra AVPL. 

Gop ftp_ses On ftp_pdu Match (addr, addr, port, port) {
    Start (ftp_cmd=USER);
    Extra (pasv_prt, pasv_addr);
};

Gop isup_leg On isup_pdu Match (m3pc, m3pc, cic) {
    Start (isup_msg=1);
    Extra (calling, called);
};

End of a Gop

Once the Pdu has been assigned to the Gop, MATE will check whether or not the Pdu matches the GopStop, if it happens, MATE will mark the Gop as stopped. Even after stopped a Gop may get assigned new Pdus matching its key unless they are GopStart. In which case it will create a new Gop starting with that Pdu. 

Gop ftp_ses On ftp_pdu Match (addr, addr, port, port) {
    Start (ftp_cmd=USER);
    Stop (ftp_cmd=QUIT); // The response to the QUIT command will be assigned to the same Gop
    Extra (pasv_prt, pasv_addr);
};

Gop dns_req On dns_pdu Match (addr, addr, dns_id) {
    Start (dns_resp=0);
    Stop (dns_resp=1);
};

Gop isup_leg On isup_pdu Match (m3pc, m3pc, cic) {
    Start (isup_msg=1); // IAM
    Stop (isup_msg=16); // RLC
    Extra (calling, called);
};

If no Stop criterium is stated for a given Gop, the Gop will be stopped as soon as it's created. However as with any other Gop Pdus matching the Gop's key will still be assigned to the Gop unless they match a Start condition in which case a new Gop using the same key will be created.

Gop's tree

For every frame containing a Pdu that belongs to a Gop MATE will create a tree for that Gop.

The example below represents the tree created by the dns_pdu and dns_req examples. 

...
mate dns_pdu:6->dns_req:1
    dns_pdu: 6
        dns_pdu time: 2.103063
        dns_pdu time since begining of Gop: 2.103063
        dns_req: 1
            dns_req Attributes
                dns_id: 36012
                addr: 10.194.4.11
                addr: 10.194.24.35
            dns_req Times
                dns_req start time: 0.000000
                dns_req hold time: 2.103063
                dns_req duration: 2.103063
            dns_req number of PDUs: 2
                Start PDU: in frame 1
                Stop PDU: in frame 6 (2.103063 : 2.103063)
        dns_pdu Attributes
            dns_resp: 1
            dns_id: 36012
            addr: 10.194.4.11
            addr: 10.194.24.35

Other than the pdu's tree this one contains information relative to the relationship between the Pdus that belong to the Gop. That way we have:

mate.xxx.Time vs mate.xxx.Duration

There are two "timers" for a Gop:

So:

Grouping Gops together (Gog)

When Gops are created or whenever its AVPL changes Gops are (re)analyzed to check if they match an existent or can create a group of groups (Gog). The Gop analysis is divided in two phases. In the first phase the still unassigned Gop is checked to verify whether it belongs to an already existing Gog or may create a new one. The second phase eventually checks the Gog and registers its keys in the gogs index.

gop_analysis.png

There are several reasons for the author to believe that this feature needs to be reimplemented, so probably there will be deep changes in the way this is done in the near future. This section of the documentation reflects the version of mate as of wireshark 0.10.9; in future releases this will change.

Declaring a Group Of Groups

The first thing we have to do configuring a Gog is to tell MATE that it exists. 

Gog web_use {
   ...
};

Telling MATE what could be a Gog member

Then we have to tell MATE what to look for a match in the candidate Gops 

Gog web_use {
    Member http_ses (host);
    Member dns_req (host)
};

Getting interesting data into the Gop

Most often, not only those attributes used for matching would be interesting. In order to copy from Gop to Gog other interesting attributes we might use Extra like we do for Gops. 

Gog web_use {
    ...
    Extra (cookie);
};

Gog's tree

mate http_pdu:4->http_req:2->http_use:1
    http_pdu: 4
        http_pdu time: 1.309847
        http_pdu time since begining of Gop: 0.218930
        http_req: 2
            ... (the gop's tree for http_req: 2) ..
        http_use: 1
            http_use Attributes
                host: www.example.com
            http_use Times
                http_use start time: 0.000000
                http_use duration: 1.309847
            number of GOPs: 3
                dns_req: 1
                    ... (the gop's tree for dns_req: 1) ..
                http_req: 1
                    ... (the gop's tree for http_req: 1) ..
                http_req of current frame: 2

We can filter on:

AVPL Transformations

Syntax

AVPL Transformations are declared in the following way: 

 Action=Transform; Name=name; Match=Strict; Mode=Insert; match_attr1=xxx; match_attr2=yyy; .replace_attr1=XXX; .replace_attr2=XXX;
 Action=Transform; Name=name; Match=Strict; Mode=Insert; match=avpl2 ...; .replace=avpl2 ...;
 Action=Transform; Name=name; Match=Strict; Mode=Insert; match=avpl3 ...; .replace=avpl3 ...;
 Action=Transform; Name=name; Match=Strict; Mode=Insert; match=avpl4 ...; .replace=avpl4 ...;

The Name is the handle to an AVPL transformation. It is used to refer to a certain sequence. Every AVPL transformation is a sequence of AVPLs that share a common Name, they will be executed in the order they appear in the config file whenever they are invoked.

The Match AVP instructs MATE which type of AVPL match should it use it can be one of Strict, Every or Loose.

The Mode AVP instruct MATE how the replace AVPL should be used; it can be either just Insert the resulting attributes into the operand AVPL or to Replace the matching attributes with the resulting attributes.

The match avpl is given using the actual names of the AVPs to be matched

The resulting AVPs are given prepending a '.' to their name.

Usage

Once declared, a transformation sequence can be added to the list of transformations for the AVPLs of PDUs, Gops or Gogs. This is done using the following commands:

Operation

transform.png

MATE's manipulation tool

MATE's Transforms can be used for many different things, like:

Multiple Start/Stop conditions for a Gop

Using Transforms we can add more than one start or stop condition to a Gop. 

Action=Transform; Name=start_cond; attr1=aaa; attr2=bbb; .msg_type=start;
Action=Transform; Name=start_cond; attr3=www; attr2=bbb; .msg_type=start;
Action=Transform; Name=start_cond; attr5^a; .msg_type=stop
Action=Transform; Name=start_cond; attr6$z; .msg_type=stop;

Action=PduDef; Name=pdu; ...
Action=PduTransform; For=pdu; Name=start_cond;

Action=GopDef; Name=gop; ...
Action=GopStart; For=gop; msg_type=start;
Action=GopStart; For=gop; msg_type=stop;

Marking Gops and Gogs to filter them easily

Action=Transform; Name=marks; addr=10.10.10.10; user=john; .john_at_host;
Action=Transform; Name=marks; addr=10.10.10.10; user=tom; .tom_at_host;

...

Action=GopDef; Name=gop; ...
Action=PduTransform; For=gop; Name=marks;

After that we can filter a capture file using mate.gop.john_at_host or mate.gop.tom_at_host

Adding direction knowledge to MATE

Action=PduDef; Name=pdu; ... src=ip.src; dst=ip.dst; addr=ip.addr; port=tcp.port; start=tcp.flags.syn; stop=tcp.flags.fin; stop=tcp.flags.rst;

Action=Transform; Name=direction; Match=Strict; Mode=Replace; src=192.168.0.2; dst=192.168.0.3; .direction=from_2_to_3;
Action=Transform; Name=direction; Match=Strict; Mode=Replace; src=192.168.0.3; dst=192.168.0.2; .direction=from_3_to_2;

Action=PduTransform; For=pdu; Name=direction;

Action=GopDef; ... addr; addr; port; port;
Action=GopExtra; ... direction;

NAT

NAT can create problems when tracing, but we can easily workarround it Transforming the nated address and the Ethernet of the router into the non-NAT address: 

Action=PduDef; Name=pdu; ... ether=ether.addr; addr=ip.addr; port=tcp.port;

Action=Transform; Name=denat; Match=Strict; Mode=Replace; addr=192.168.0.5; ether=01:02:03:04:05:06; .addr=123.45.67.89;
Action=Transform; Name=denat; Match=Strict; Mode=Replace; addr=192.168.0.6; ether=01:02:03:04:05:06; .addr=123.45.67.90;
Action=Transform; Name=denat; Match=Strict; Mode=Replace; addr=192.168.0.7; ether=01:02:03:04:05:06; .addr=123.45.67.91;

Action=PduTransform; For=pdu; Name=denat;

About MATE

MATE was originally written by Luis Ontanon a Telecomunications systems troubleshooter as a way to save time filtering out the packets of a single call from huge capture files using just the calling number. Later he used the time he had saved in making it flexible enough to work with protocols other than the ones he was directly involved with.

Mate/Manual (last edited 2012-01-15 17:41:54 by BillMeier)