chrismattmann / lucene-geo-gazetteer

Uses Apache Lucene, OpenNLP and geonames and extracts locations from text and geocodes them.

Lucene Geo-Gazetteer

A command line gazetteer built around the Geonames.org dataset, that uses the Apache Lucene library to create a searchable gazetteer.

Use

The Geonames.org dataset contains over 10,000,000 geographical names corresponding to over 7,500,000 unique features. Beyond names of places in various languages, data stored include latitude, longitude, elevation, population, administrative subdivision and postal codes. All coordinates use the World Geodetic System 1984 (WGS84).

1. What we need here is to download the latest version of allCountries.zip file from GeoNames.org: curl -O http://download.geonames.org/export/dump/allCountries.zip
2. Unzip the GeoNames file: unzip allCountries.zip
3. Take the allCountries.txt and use it to create a geoIndex: java -cp target/lucene-geo-gazetteer-<version>-jar-with-dependencies.jar edu.usc.ir.geo.gazetteer.GeoNameResolver -i geoIndex -b allCountries.txt
4. Then search the index (e.g., for Pasadena and Texas): java -cp target/lucene-geo-gazetteer-<version>-jar-with-dependencies.jar edu.usc.ir.geo.gazetteer.GeoNameResolver -i geoIndex -s Pasadena Texas
5. The service mode:

        #Launch Server
        $ lucene-geo-gazetteer -server
        # Query
        $ curl "localhost:8765/api/search?s=Pasadena&s=Texas&c=2"

Questions, comments?

Send them to Chris A. Mattmann.

Contributors

• Thamme Gowda N., USC
• Madhav Sharan, USC
• Yun Li, USC
• Chris A. Mattmann, JPL
• Maziyar Boustani, JPL

Credits

This project began as the CSCI 572 project of Yun Li on the NSF Polar CyberInfrastructure project at USC under the supervision of Chris Mattmann. You can find Yun's original code base here.

This work was sponsored by the National Science Foundation under funded projects PLR-1348450 and PLR-144562.

License

Apache License, version 2

lucene-geo-gazetteer's Issues

Fix output format.

The current JSON output format is problematic and not intuitive.

For example,
$ lucene-geo-gazetteer -s "Los Angeles" -c 5 search produces the following result:

[  
  {  
    "Los Angeles":[  
      "Los Angeles County",
      "-118.26102",
      "34.19801",
      "US",
      "CA",
      "037",
      "City of Los Angeles",
      "-118.53995",
      "34.15649",
      "US",
      "CA",
      "037",
      "City Lands of Los Angeles",
      "-118.23035",
      "34.05807",
      "US",
      "CA",
      "037",
      "Los Angeles",
      "-79.3643",
      "-1.41917",
      "EC",
      "13",
      "1207",
      "Los Angeles",
      "-72.32774",
      "-37.40792",
      "CL",
      "06",
      "83"
    ]
  }
]

---

Desired and Intuitive output should have been:

[  
  {  
    "Los Angeles":[  
      [  
        "Los Angeles County",
        "-118.26102",
        "34.19801",
        "US",
        "CA",
        "037"
      ],
      [  
        "City of Los Angeles",
        "-118.53995",
        "34.15649",
        "US",
        "CA",
        "037"
      ],
      [  
        "City Lands of Los Angeles",
        "-118.23035",
        "34.05807",
        "US",
        "CA",
        "037"
      ],
      [  
        "Los Angeles",
        "-79.3643",
        "-1.41917",
        "EC",
        "13",
        "1207"
      ],
      [  
        "Los Angeles",
        "-72.32774",
        "-37.40792",
        "CL",
        "06",
        "83"
      ]
    ]
  }
]

This leaves room for future improvements without breaking the existing clients. Try adding a new field to output and guess what will happen array index?

---

Better yet:

[  
  {  
    "Los Angeles":[  
      {  
        "name":"Los Angeles County",
        "lon":-118.26102,
        "lat":34.19801,
        "country":"US",
        "state":"CA",
        "stateid":"037"
      },
      {...}
     ]
   }
]

This is definitely better, because it is easy to consume by rest clients.

Add a testing suite for location relevancy

Test suite must have diverse test cases covering continents, countries, states, cities, towns etc..

Consider using https://travis-ci.org/

Once we have this suite we can ensure future modifications don't disturb current classifiers.

Return additional fields along with co-ordinates of location

@chrismattmann : How about returning country code, state, county along with co-ordinates? This information might make sense when we analyse this data in a context. It will allow us to group locations based on location's country, state and county.

[
{"Pasadena" : [
"Pasadena",
"-74.06446",
"4.6964",
"US",
"CA",
"37"
]}
]

Use sniffer plugin to disable jdk 8 API

With current setup we enforce jdk 7 for building project. It will undoubtedly compile fine with jdk 8 but the idea behind enforcing 7 was to not depend on 8 entirely as some API in jdk 8 will not compile with java

With sniffer we can ensure that code does not include any API which are not included in java 7 even though it's compiled by jdk 8

Thoughts? @chrismattmann

            <plugin>
                <groupId>org.codehaus.mojo</groupId>
                <artifactId>animal-sniffer-maven-plugin</artifactId>
                <version>1.7</version>
                <executions>
                    <execution>
                        <id>signature-check</id>
                        <phase>compile</phase>
                        <goals>
                            <goal>check</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <signature>
                        <groupId>org.codehaus.mojo.signature</groupId>
                        <artifactId>java17</artifactId>
                        <version>1.0</version>
                    </signature>
                </configuration>
            </plugin>

Add more fields to lucene index to increase result relevancy

Fields "feature class", "feature code", "population","country code", "admin1 code", "admin2 code" can help us defining granularity of our locations. Then we will use these fields to select most relevant location for a String.

1. "feature class" is a set of categories to bucket land masses in a broader class
2. "feature code" is a unique category to identify it as a city or state, country.
3. "population" have estimates of people living in that location
4. "country code" Country of location
5. "admin1 code" Can be State of location
6. "admin2 code" Can be county/district

For example Pasadena currently returns coordinates - 4.6964,-74.06446 which point to some Pasadena in Columbia while more known location could be Pasadena CA / Pasadena TX.

This link contain all codes: http://www.geonames.org/export/codes.html.
This link contain schema and other deatils on data set http://download.geonames.org/export/dump/readme.txt

@chrismattmann Any more / Any less field you suggest?

Strange search results (esp. with lowercase names).

This is perhaps not a bug, but just want to share some examples of incorrect matching of locations:

$ ./src/main/bin/lucene-geo-gazetteer -s "russia" "china" "America"
[
{"russia" : [
"Republic of Belarus","28.0","53.0","BY","00",""
]},
{"china" : [
"Taiwan","121.0","24.0","TW","00",""
]},
{"America" : [
"South America","-57.65625","-14.60485","","",""
]}
]

Russia and China are recognized correctly if capitalized.

Use new fields in lucene index to calculate most relevant locations

One Possible implementation:

1. Define a sorting criteria for "feature class" and sort on it. This could be a possible order A P S T L H R V U
2. For each "feature class" define sorting criteria of "feature code" and sort on it.
3. Now sort it by population within same "feature class" and "feature code"
4. We need to modify our edit distance calculations because "China" is stored with "People's Republic of China" and so on. I propose below based on my observation on "alternatenames" field.
   • "alternatenames" contains a CSV of all pronunciations and synonyms for that location.
   • We split "alternatenames" and calculate edit distance with each of the variable on our sorted list
     • if it's a exact match add this location to result list
     • if not store all edit distances and assign a weight to all "feature code" to prioritize bigger land masses and add those in the end of results.

@chrismattmann
Can we take one extra parameter to define number of results that should be returned by lucene-geo-gazetteer?

Adding some query time controls

1. Add an option to initialize ranking weights with population value to give priority to population.
2. Add an option for wildcard search.
3. Add an option to give equal weight to alternative name occurrence.
4. Add an option to disable Ranking algorithm, relying only on population.

Patch for missing names in allCountries.txt

Locations in allCountries.txt are present with their official names. There are few locations which are more known by their popular names all over the world than their official names.

Example -

• "Argentine Republic" as Argentina
• "Repubblica Italiana" as Italy

Possible way - Use country code list in geonames.org [0] and add a dummy document in lucene index for all PCLI.

[0] - http://download.geonames.org/export/dump/countryInfo.txt

Reverse geocoding using lucene-spatial

Aim - For a given latitude and longitude return name of city, state and country.

Lucene API - https://lucene.apache.org/core/5_0_0/spatial/index.html