These tutorials assume that you have access to ArcGIS either in our computer lab or installed on your computer so there won't be any instructions on how to install. They are written for someone with basic knowledge of ArcGIS's interface, that wants a walkthrough how this tool can be used to help them ask and answer their spatial research questions.
For tutorials starting at a beginner level I recommend those on ESRI's website or if you have a New York Public Library card, they have several great tutorials for newcomers to ArcGIS on their Lynda.com portal.
The data I'll be using in these tutorials comes from felony drug arrest statistics by county that I got from the New York State Division of Criminal Justice Services, spatial data from the New York State GIS Portal and data from the Law Enforcement Support office at the Defense Logistics Agency concerning the 1033 program. In brief, the 1033 program lets local law enforcement agencies request decommissioned military equipment ranging from office furniture, to night-vision, to weapons to Mine-Resistant Ambush Proof Vehicles (MRAPs).
While the data I'm using is real from a project that I did, it isn't going to be updated so please go back to the original sources mentioned if you'd like to explore it further. I picked these data sets because they were handy and they are large and complex enough to show how to work with different kinds of data but not too complicated to get in the way of explaining the process. The maps made in these examples are not necessarily scholarly rigorous but hopefully the process of making them will get you familiar enough with ArcGIS Pro so that when you map your own data, you'll have more time for scholarly rigor!
Sometimes you'll be dealing with more granular data. Instead of a whole city, you want to map a particular address within that city, instead of a whole national park, the peak of a mountain within that park. This is point data. This can be in a shapefile of its own, in which case you'd just drag it on the canvas the same as any other shapefile. It can arrive in table form as well, as a list of addresses or as latitude and longitude coordinates, either as secondary data, or primary data you've recorded yourself with a GPS tracker.
In this exercise, you'll learn how to take a csv file with latitude and longitude coordinates, and use the Display X,Y function to add them to the map. You'll see how to use Google Maps to find the latitude and longitude for an address to revise any coordinates you may have gotten wrong and where those coordinates will be visible when your web-browser is showing information for that location.
The files below are what will be used for this exercise. One is a list of police stations' latitudes and longitudes, geocoded-police-stations.csv that I got using Google's geocoding API and a python script. The last tab in this box is a tutorial describing how I did that, but that's out of the scope of this exercise. The other is the base map created in an earlier exercise, New York State Base Map.mxd
A geographic coordinate system tells ArcMap how to plot the data on a map or globe. If you have one coordinate system for your basemap and another for the points you are plotting, then the points will be displayed inaccurately. So, to plot the data from geocoded-police-stations.csv accurately, you'll need to find out what coordinate system is being used in the data frame, which is the canvas you're working on.
This is because I got the latitude and longitude data from these places via Google's Map API which I queried using a script and I didn't check all of the places before putting them on my map. Google's first guess for some of these places must be incorrect. Fortunately, now that I know which ones are very wrong, I can just update the csv file with the correct information for those places and repeat this procedure with the corrected sheet.
The first number is the latitude and the second is longitude.
That will change the map to the below, where all the blue star symbols are now within the boundaries of New York.
It'll just give you a default symbol for these events, but you can doubleclick on the symbol to change what it is. This opens the Layer Properties window, and you can change the symbol in the Symbology tab. I'm making it a larger blue star.
I've attached my final map below if you want to see how I configured it.
You will sometimes collect location data in the form of addresses rather than as latitude and longitude. In this case, your data will need the extra step of geocoding before you can use it in ArcGIS. Geocoding takes a location like an address or city or state and renders it as latitude and longitude coordinates so that ArcGIS or other mapping programs can read it and mark its location.
In this exercise, you'll take a list of addresses and use the US Census Bureau's Geocoding service in order to batch process a group of addresses and get their coordinates. You'll read the output file that you receive to determine if the results you got were accurate and to take note of the addresses that the service was unable to provide. You'll find the coordinates that the gecoding service was unable to find matches for by using Google Maps. Finally you'll plot the addresses that you have found using the Display XY Data function in ArcGIS.
It's important to note, however, that this geocoding service only works with U.S. addresses so if you are looking for addresses in foreign countries, search their government's websites to see if they provide a similar service. Additionally, it has a limit of 10,000 records so if you need more than that, you'll have to divide up your searches.
Below I've attached the csv file that you'll be using with on the US Census Bureau Geocoding website, and the map document that you'll be plotting these addresses in once you've received your coordinates.
The amount of addresses that this geocoder will give you will vary, but at least some of them aren't going to have matches depending on the kind of data you are looking for. The site acknowledges that it is better with residential vs. commercial addresses. However in this case there are only about 6 addresses out of 26 that didn't have matches, so it did have the majority. When the number is that low, it will be easy to just use Google Maps to fill in this information manually. You wouldn't want to do this for a large chunk of your data however as it is time-consuming. You could also try running it again, since I have gotten more matches a second time through than the first, but if this few records are missing matches, it will take less time to just use Google Maps.
You probably use Google Maps quite a bit and didn't realize you were looking at coordinate data. But in fact, whenever you look up an address in Google Maps, the latitude and longitude will be right there for you in the URL. The first number after the @ is the latitude, and the second is the longitude.
You're now set to plot your results in ArcMap
Open New York State Base Map in ArcMap.This will put a base map of New York on your canvas to give you context of where the points you are plotting are located.
Using the Catalog window, navigate to the folder where you have saved GecodedResultsForArcMap.csv and drag it onto the canvas. Your map won't change since it is a table and not a shapefile, but the view in your Table of Contents window will change to the List By Source view and you'll see the file and its path displayed.
In the previous step, you added Longitude and Latitude information to your list of addresses, and this is sufficient information for ArcMap to add a point at every coordinate spot you provide that information for, however it will need to know which coordinate system to use to do so. You'll first need to figure out what the coordinate system for your data frame is so the locations won't be distorted.
This exercise will assume that you already have Python installed on your computer and that you have some experience using that language to query an API. I strongly recommend Codecademy or the API exercise on Programming Historian if either of those are not the case. I'll be explaining how to write the script that gets me this information, but I won't be explaining how to use Python or APIs in general.
The tutorial here walks you through how I got the list that I am now using in the Plotting with Longitude and Latitude Data exercise, the list of coordinates for each of the police stations that I am mapping. I generated this using just the list of the police stations' names, the Google API and a simple Python script.
Credit here to Fred Gibbs whose script on his blog I barely had to tweak in order to write my script
This is a relatively simple script that just accesses Google Map's API, feeds it a list of names, and writes the responses that Google sends back (the longitude and latitude) into both the Python console so you can keep track of what progress you're making, and writes them to a csv file that you can later use to enter these coordinates into ArcMap.
import requests
import csv
import time
'requests' is the module that allows you to make request from APIs online, and to allow Python to read information on websites, as well as convert jason material into Python objects.
'csv' is the module that lets Python interface with csv files including writing to them.
'time' lets you give instructions to Python about how much time to allot between tasks. Since you're querying an API over and over again, you'll want to give a pause in between requests so you're not putting too much strain on your servers and they hopefully won't try to kick you off because of that.
I'll be using the variable inputfile for the former and the variable outputfile for the latter.
inputfile = open('police-placelist.txt','r')
outputfile = csv.writer(open('geocoded-police-stations.csv','w'))
The first line is saying that when the variable inputfile is used later in the script, it's referring to the content in the file that has all your the law enforcement agency names - (police-placelist.txt) when it is read (r). Make sure you have this file in the same folder that you are saving the script to so that it can find it.
The second line is saying that the variable outputfile is the file geocoded-police-stations.csv being written to (w).
for row in inputfile:
row = row.rstrip()
url = 'http://maps.googleapis.com/maps/api/geocode/json'
payload = {'address':row, 'sensor':'false'}
You're first telling Python that every line in your inputfile will be called by the variable row, and then giving it a set of instructions to execute for that particular row, like r.strip() which will remove all the extra white space at the end of the row.
Next you feed in the url for the Google Maps API and assign it to the variable 'url' which you will refer to later.
Then you create a variable 'payload' which corresponds with a dictionary with two bits of information that the Google API will ask for: address which you're telling it to feed the API the police station name from your file (row), and sensor, which is just a metric it requires that needs to be set to true or false to function.
try:
r = requests.get(url, params=payload)
json = r.json()
lat = json['results'][0]['geometry']['location']['lat']
lng = json['results'][0]['geometry']['location']['lng']
newrow = [row,lat,lng]
The beginning of the function that says 'try' means that this is what you're telling the script to perform if it's possible ('except' and 'else' later in the code will tell the script what to do if what you've asked it to do isn't possible)
r is a variable you're creating that just consists of Python requesting the url you've given it followed by the parameters you've given it above in the variable 'payload' (the name of the police station and the sensor being set to 'false').
Make a variable of json that is the response of your request to that API ('r') being read as a json script.
From the whole json script the API will deliver as a response, you care about two things: the latitude and the longitude. You'll two variables to correspond with those attributes 'lat' and 'lng'.
If you actually open the JSON script for a result from the Google Maps API query, it looks something like this
{
"results" : [
{
"address_components" : [
{
"long_name" : "Albuquerque",
"short_name" : "Albuquerque",
"types" : [ "locality", "political" ]
},
{
"long_name" : "Bernalillo County",
"short_name" : "Bernalillo County",
"types" : [ "administrative_area_level_2", "political" ]
},
{
"long_name" : "New Mexico",
"short_name" : "NM",
"types" : [ "administrative_area_level_1", "political" ]
},
{
"long_name" : "United States",
"short_name" : "US",
"types" : [ "country", "political" ]
}
],
"formatted_address" : "Albuquerque, NM, USA",
"geometry" : {
"bounds" : {
"northeast" : {
"lat" : 35.2180539,
"lng" : -106.4711629
},
"southwest" : {
"lat" : 34.9467659,
"lng" : -106.881796
}
},
"location" : {
"lat" : 35.110703,
"lng" : -106.609991
},
"location_type" : "APPROXIMATE",
"viewport" : {
"northeast" : {
"lat" : 35.2180539,
"lng" : -106.4711629
},
"southwest" : {
"lat" : 34.9467659,
"lng" : -106.881796
}
}
},
"types" : [ "locality", "political" ]
}
],
"status" : "OK"
}
That's why you're telling it to check JSON for:
"location" : {
"lat" : 35.110703,
"lng" : -106.609991
}
and assigning those to the variables of lat and lng.
Last in this pack of text is the variable that refers to the information that you ultimately want to be written to the file you're creating, newrow
newrow = [row,lat,lng]
The lines that you're telling it to write to the csv is that for every place name that it gets from the txt file, to write a line to the csv with that place name, then the latitude, then the longitude.
print newrow
outputfile.writerow(newrow)
time.sleep(1)
print newrow will print your results to the console window. This isn't necessary but is a good way for you to test that your script is working, and will give you a backup to consult if writing it to the file fails for some reason.
outputfile.writerow(newrow) tells it to write that variable 'newrow' (the place name, the latitude and the longitude) to your output csv file that you specified way up there at the top.
time.sleep(1) is just telling the script to wait one second before your next request to Google Maps API. If you don't include something like this it will just do it as quickly as the code executes, and the server you're requesting information from might decide to kick you off for taking up too much of its bandwidth.
except IndexError:
newrow = [row,'null','null']
print newrow
This is telling your code, if you get an IndexError, to write out the name of the place, and for the next fields just write 'null' both to your CSV file and to the console. You can then look up the results for these (hopefully few) place names later for yourself manually. I used IndexError because that was the error I tended to get that stopped my code before I added in try and except
One more function completes this, 'else'. This tell your script what to do if it can't execute the code you put in under try but also doesn't run into the exception you put forth in except.
else:
newrow = [row,'null','null']
This is just telling your script that if it can't execute the code, and the reason isn't the one you gave it in except, then it should still write a newrow in your csv that consists of the place name, and for the latitude and longitude columns it should write null for each of them.
I'll attach my code below if you want to see it as I wrote it, but if you've been following along in your own script editor you should be good to execute the code now.
Caveat: even with the try and except your code might not get through the whole list before hitting something that makes it stop. I advise in that case, instead of running the whole list again -
outputfile = csv.writer(open('geocoded-police-stations.csv','w'))
to be
outputfile = csv.writer(open('geocoded-police-stations.csv','a'))
with 'w' for write switched to 'a' for append so that it doesn't overwrite the file that you've already created.
The script should go through your list in order, so it will be easy to tell where it left off. Then you can just merge each of the csv files you get with the latitude and longitude information into one file that contains all your data. Make sure and double check to make sure that nothing is missing. Some may not be findable on the first pass and you'll need to feed through those addresses again.
Proceed to the first exercise of this page: 'Plotting with Latitude and Longitude Data' to see how to add this data to the map.