Inheriting base entities from maltego.entities about canari

[allfro]

Hi Matt,

Built-in entities are automatically excluded when running canari generate-entities unless you add the --maltego-entities to the command. Help on any command can be done by typing canari <command> help, like so:

canari generate-entities --help
usage: canari generate-entities [output file] [options]

Converts Maltego entity definition files to Canari python classes. Excludes
Maltego built-in entities.

positional arguments:
  [output file]         Which file to write the output to.

optional arguments:
  -h, --help            show this help message and exit
  --mtz-file <mtzfile>, -m <mtzfile>
                        A *.mtz file containing an export of Maltego entities.
  --exclude-namespace <namespace>, -e <namespace>
                        Name of Maltego entity namespace to ignore. Can be
                        defined multiple times.
  --namespace <namespace>, -n <namespace>
                        Name of Maltego entity namespace to generate entity
                        classes for. Can be defined multiple times.
  --maltego-entities, -M
                        Generate entities belonging to the 'maltego'
                        namespace.
  --append, -a          Whether or not to append to the existing *.py file.
  --entity <entity>, -E <entity>
                        Name of Maltego entity to generate Canari python class
                        for.

Also you can always inherit and extend already built-in entities by just sub-classing them, like so:

from canari.maltego.entities import IPv4Address

@EntityField(name='blah.prop', propname='prop')
class MyEntity(IPv4Address):
     namespace='blah'

But what I would do if I were in your shoes is create a Dynamic property in the transform like this:

from canari.maltego.entities import IPv4Address

def dotransform(request, response):
    e = IPv4Address('1.1.1.1')
    e += Field('name', 'value', displayname='My Name')
    response += e
    return response

[mattnewham]

Hi Nadeem

This is awesome, thanks a lot for clarifying. I thought it best to put these questions into github so I/we can keep track of things and also anybody else can come across the answers. If you want me to document any of these advanced features (like the link labels etc) I can start writing things up and send you it?

Matt

[mattnewham]

Just a slight change to your above suggestion - need to import Field from canari.maltego.message:

from canari.maltego.entities import IPv4Address
from canari.maltego.message import Field

def dotransform(request, response):
    e = IPv4Address('1.1.1.1')
    e += Field('name', 'value', displayname='My Name')
    response += e
    return response

[mattnewham]

Also, I couldn't get the inheritance to work. It just behaved as if I hadn't put (IPv4Address) after my entity type...

[allfro]

Could you post a code sample?

[mattnewham]

OK, so this is my transform to query my netflow data which creates the entity NetFlowDST:

def dotransform(request, response):
    """
    TODO: write your data mining logic below.
    """
#Gonna run the netflowquery function in mongodbquery with a query of srcaddr(our input ip from maltego)
    values = mongodbquery.netflowquery('srcaddr','%s' % request.value)

    processed = []

    for i in values:
        if i['_id'] not in processed:
            response += NetFlowDST(
                i['dstaddr'],
                weight = i['#:doctets'],
                dstport = i['dstport']
            )
            processed.append(i['_id'])

    return response

This works fine, and returns the following (sample):

    `- Entity:  {'Type': 'ctxfx.NetFlowDST'}
      `- Value: 173.194.78.94
      `- Weight: 161
      `- AdditionalFields:
        `- Field: 443 {'DisplayName': 'Destination Port', 'Name': 'ctxfx.dstport', 'MatchingRule': 'strict'}

Here is my entity type:

@EntityLinkField(name='maltego.link.label', propname='linklabel', matchingrule=MatchingRule.Loose)
@EntityField(name='ctxfx.dstport', propname='dstport', displayname='Destination Port')
class NetFlowDST(IPv4Address):
    namespace='ctxfx'
    pass

[allfro]

Hi @mattnewham,

Just a point of clarification here: You do not need to define the @EntityLinkField because this is inherited automatically by all entity types.

Question: What isn't working? :)

[mattnewham]

Yeah I figured that our then forgot it was in the code I posted. Link
labels are working just great now.

I managed to get dynamic fields onto existing entities working well now.

Only other thing I was trying was making my own entity and inheriting the
ipv4address base entity type. When I changed my entity to be ipv4address
inside the brackets as your example stated, nothing changed. I might have
misunderstood how to do it, I'm often pretty slow!
On 25 Feb 2013 22:13, "allfro" [email protected] wrote:

Point of clarification,

You do not need to define the @EntityLinkField because this is inherited
automatically by all entity types.

Question,

What isn't working? :)

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3#issuecomment-14079386.

[allfro]

What change were you expecting?

[mattnewham]

Expecting it to be an ipv4address entity in maltego . When I create my own
entity type from within maltego and inherit the ipv4address base entity, it
behaves just as an IP address entity but I can make it look however I want
etc. That's what I want to be able to do but I'm too stupid to figure it
out!
On 25 Feb 2013 22:38, "allfro" [email protected] wrote:

What change were you expecting?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3#issuecomment-14080755.

[allfro]

Ah,

No, then what you want is not to inherit an IPv4Address but to use an IPv4Address entity and add dynamic fields to it. Entity definitions always take on the name of the class it's defined in. Inheritance will only save you from redefining the entity fields that are already in another entity. That's all it's meant for :) So your code should look like this:

def dotransform(request, response):
    """
    TODO: write your data mining logic below.
    """
#Gonna run the netflowquery function in mongodbquery with a query of srcaddr(our input ip from maltego)
    values = mongodbquery.netflowquery('srcaddr','%s' % request.value)

    processed = []

    for i in values:
        if i['_id'] not in processed:
            e = IPv4Address(
                i['dstaddr'],
                weight = i['#:doctets']
            )
            e += Field('dstport', i['dstport'])
            response += e
            processed.append(i['_id'])

    return response

[mattnewham]

That's what I think I have done now. Seems to work well. What exactly is
happening in maltego then if I create my own entity but inherit
ipv4address? What's its intended function if I can achieve the same results
by adding dynamic fields to an existing maltego entity type? Or is that
basically what is going on when I make my own in maltego?

The more I use canari the better it gets. Its so flexible!

I per ordered my leap motion now after you showed me it!
On 25 Feb 2013 22:52, "allfro" [email protected] wrote:

Ah,

No, then what you want is not to inherit an IPv4Address but to use an
IPv4Address entity and add dynamic fields to it. Entity definitions
always take on the name of the class it's defined in. Inheritance will only
save you from redefining the entity fields that are already in another
entity. That's all it's meant for :)

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3#issuecomment-14081436.

[allfro]

From what I can tell, inheritance in Maltego is used to simplify transform association to entity types. For instance, look at all the DNS related entities. They are all subclasses of DNSName. If you right click on any of them, you will see the same types of transforms apply on the child and parent entities. That being said, you could always use inheritance if you want to specify a custom IPv4Address entity type but inherit all of the IPv4Address type's transforms. It will be called something else though :)

[mattnewham]

That's really what I was trying to get my entity to do, I wanted my results
as my own entity but to have the ipaddress transforms available to them.

I still don't get how to inherit entities in canari though, I'm doing
something wrong, clearly!

Its no really an issue ATM but something we've been toying with at work is
"branding" our entities so will become more applicable when we start doing
that
On 25 Feb 2013 23:36, "allfro" [email protected] wrote:

From what I can tell, inheritance in Maltego is used to simplify transform
association to entity types. For instance, look at all the DNS related
entities. They are all subclasses of DNSName. If you right click on any of
them, you will see the same types of transforms apply on the child and
parent entities. That being said, you could always use inheritance if you
want to specify a custom IPv4Address entity type but inherit all of the
IPv4Address type's transforms. It will be called something else though :)

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3#issuecomment-14083487
.

[allfro]

So one thing to note is that entity inheritance in Canari is different than entity inheritance in maltego. Make sure you apply inheritance when you design the entity in Maltego.

[allfro]

Hi @mattnewham,

Can I close this thread?