diff --git a/.gitignore b/.gitignore index 50a89a0..2356f54 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,4 @@ -*.js -data/ -data.*/ +*.json +cache/ +.vscode +tba_token.txt diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..bf830be --- /dev/null +++ b/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2020 Ethan Shaw + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/README b/README deleted file mode 100644 index c2a4f16..0000000 --- a/README +++ /dev/null @@ -1,62 +0,0 @@ -Readme file for FIRSTmap scraper #2 - -This scraper is likely to be a high-maintenance item, as there are a number - of hacks to work around incomplete team info from thebluealliance, and - differences between place names in different databases. To use this, - copy/extract/clone the files to a directory, then run "./get_all_data" - -Last updated by Gus "GeeTwo" Michel II, 21 Oct 2017 - -README This file - -ask_google script to get a lat/lon from a "place" file. - Usage: ./ask_google >> geo_cache - This file must be run manually; it is not run from get_all_data - -geo_cache pipe delimited file || - this file is manually built from previous calls to ask_google - and other sources - -attribs A list of the team attributes on V3 of thebluealliance API - Used by build_teamInfo - -build_teamInfo Builds teams.js from merged.js and georef data - Automatically called directly from get_all_data - -get_all_data The main script which runs the others (except ask_google) - Gets postal data, event data, and team data from the web and - builds comps.js and teamInfo.js. Stores intermediate - results in data subdirectory, which it creates - -get_events gets the events list from thebluealliance and builds - events.js. Uses lat and lng fields from TBA. - Automatically called directly from get_all_data - -get_lists gets team lists from thebluealliance.com. - Automatically called directly from get_all_data - -get_postal Gets geocoding data from the web and formats some of it. - These data files go in the data subdirectory and are used - by build_teamInfo - Automatically called directly from get_all_data - -make_latlng extracts the lat/lon from a google place file. - called from ask_google - -merge_lists merges the team "page" lists into one file - Automatically called directly from get_all_data - -TBA-auth A file containing a TBA authorization string. - This is not in the repository, as each user is - expected to get his/her own. - Used by get_lists and get_events. - -unicodes.ascii Conversion table from unicodes found in team - place names to ascii characters; drops - diacritical marks and such. - Used by build_teamInfo - -YEAR The current year for purposes of extracting team - lists and event list from thebluealliance.com - Used by get_lists and merge_lists. - diff --git a/README.md b/README.md new file mode 100644 index 0000000..de12270 --- /dev/null +++ b/README.md @@ -0,0 +1,64 @@ +FIRSTMap Scraper +=== + +This scraper is likely to be a high-maintenance item, as there are a number +of hacks to work around incomplete team info from thebluealliance and +differences between place names in different databases. This scraper makes use +of [FRC 1418's tbapy](https://github.com/frc1418/tbapy), a Python wrapper for +The Blue Alliance API. Location data for most places is downloaded from +GeoNames, but manually entered data comes from Google Maps. + +The scraper was built on Python 3.8. + +## Usage: +1. Copy/extract/clone the files to a directory. +2. Install the tbapy library (`pip3 install tbapy`). +3. Get a Read API Key from TheBlueAlliance.com/account and put it in a file + called `tba_token.txt`. +3. Run `python scraper.py ` (assuming you already have Python 3 installed; if + not, install it). +4. Manually find any places that could not be found (such as by using + `ask_google`, which works some of the time). + +If the program has to be run multiple times in a row (such as to make +adjustments to it, correct issues, etc.), the program can be called with the +`usecache` argument to reuse the already-downloaded GeoNames files instead of +taking the time to redownload them every time. Example usage: +`python scraper.py usecache` + +The scraper creates a `teams.json` file, which goes into the `data` +directory of FIRSTMap. This file contains teams' latitude and longitude +coordinates. The file `teamFullInfo.json` is also created, which, in +addition to locations, also contains the attributes listed in the +scraper's `TEAM_ATTRIBS` constant. For more detailed documentation of +how the scraper works, consult the comments and code in +(scraper.py)[scraper.py]. + +## Descriptions of Files + +**[scraper.py](scraper.py)** - The scraper python script, which gets the postal +code data, location data, and team data from GeoNames and TheBlueAlliance and +uses the data to create the teams.json and teamFullInfo.json. Intermediate +downloaded data is stored in the data subdirectory, which is created by the +script if it does not exist. Comments within the file explain the process in +more detail. + +**[ask_google](ask_google)** - Script to get a lat/lon from the +`data/broken_places` file.
+    Usage: `./ask_google >> geo_cache`
+    This file must be run manually; it is not run from
+    scraper.py. It works some of the time. + +**[geo_cache](geo_cache)** - A pipe delimited file where each line os of the +format `||`. This file is manually built +from previous calls to ask_google and other sources. + +**[make_latlng](make_latlng)** - Extracts the lat/lon from a google place file. +Called from ask_google. + +**tba_token.txt** - A file containing a TheBlueAlliance (TBA) +authorization token. This is not in the repository, but is required for the +scraper to function. The user is expected to get a token from TheBlueAlliance. +A logged-in TBA user can create a Read API Key from the Account page. It simply +needs to be pasted into a file called `tba_token.txt`. This is used by the +scraper to access TheBlueAlliance API. diff --git a/TBA-auth b/TBA-auth deleted file mode 100644 index 816d48d..0000000 --- a/TBA-auth +++ /dev/null @@ -1 +0,0 @@ -CK0djLdoz5ZYTypym32fp9pRdMTmkzkQkNRNgu9KFHZJZF6ovhpsYwUdfro6jCm5 diff --git a/YEAR b/YEAR deleted file mode 100644 index 145262f..0000000 --- a/YEAR +++ /dev/null @@ -1 +0,0 @@ -2020 \ No newline at end of file diff --git a/ask_google b/ask_google index 07cffb8..94f256d 100755 --- a/ask_google +++ b/ask_google @@ -2,13 +2,10 @@ # ask_google # Gets lats and lons for places in the places file. Sleeps a randomized # time between calls to avoid robot blockage -awk -F\0 ' - { - system("wget -Odata/google " \ - "\"https://www.google.com/maps/place/" $0 "\""); - "awk -F, -f make_latlng data/google" | getline lng; - "awk -F, -f make_latlng data/google" | getline lat; - close("awk -F, -f make_latlng data/google"); - printf "%s|%.3f|%.3f\n",$0,lat,lng; - system(sprintf("sleep %d",3+10*rand())); - }' data/places + +while IFS="" read -r place || [ -n "$place" ]; do + printf '%s|' "$place" + # The particular string we search for in the result was found by manual inspection + echo $(curl --get --data-urlencode "q=$place" "https://www.google.com/maps" --silent | sed -nr 's/.*\/@(-{0,1}[0-9\]+.[0-9]+),(-{0,1}[0-9]+.[0-9]+).*/\1|\2/p') + sleep $((9 + $RANDOM % 100)) +done < cache/broken_places diff --git a/attribs b/attribs deleted file mode 100644 index acc2251..0000000 --- a/attribs +++ /dev/null @@ -1,18 +0,0 @@ -address -city -country -gmaps_place_id -gmaps_url -home_championship -key -lat -lng -location_name -motto -name -nickname -postal_code -rookie_year -state_prov -team_number -website diff --git a/build_teamInfo b/build_teamInfo deleted file mode 100755 index ea34f40..0000000 --- a/build_teamInfo +++ /dev/null @@ -1,200 +0,0 @@ -#!/bin/bash -# build_teamInfo -(gawk -F'"' ' - BEGIN { - while (getline att<"attribs") {blank[att]="";}; - fields = asorti(blank, idx); - while (getline code < "unicodes.ascii") { - split(code,fld," "); - unicode[toupper(fld[1])]=toupper(fld[2]); - } -#for (a in unicode) print a, unicode[a] > "unicode-dump" - while (getline code < "data/countryInfo.txt") { - if(split(code,fld,"\t") > 4) { - ccode[fld[5]]=fld[1]; - } - } - ccode["Chinese Taipei"]="TW"; - ccode["Czech Republic"]="CZ"; - delete ccode[""]; -#for (a in ccode) print a, ccode[a] > "ccode-dump" - while (getline zip < "data/allCountries") { - split(zip,fld,"\t"); - fld[1]=toupper(fld[1]); - fld[2]=toupper(fld[2]); - ziplats[fld[1] "|" fld[2]]=fld[10]; - ziplngs[fld[1] "|" fld[2]]=fld[11]; - } -#for (a in ziplats) print a, ziplats[a] > "ziplats-dump" -#for (a in ziplngs) print a, ziplngs[a] > "ziplngs-dump" - while (getline adm < "data/admin1CodesASCII.txt") { - split(adm,fld,"\t"); - adms[fld[1]]=toupper(fld[3]); - } -#for (a in adms) print a, adms[a] > "adms-dump" - while (getline citi < "data/cities1000") { - split(citi,fld,"\t"); - fld[3]=toupper(fld[3]); - adm = adms[fld[9] "." fld[11]] - citilats[fld[9] "|" adm "|" fld[3]]=fld[5]; - citilngs[fld[9] "|" adm "|" fld[3]]=fld[6]; - if (fld[9]=="TW") { - citilats[fld[9] "|" fld[3] "|" fld[3]]=fld[5]; - citilngs[fld[9] "|" fld[3] "|" fld[3]]=fld[6]; - } else if (fld[9]=="IL") { - # Israel has bad districts and alternate city names - split(toupper(fld[4]),alt,","); - for (i in alt) { - citilats["IL|IL|" alt[i]]=fld[5]; - citilngs["IL|IL|" alt[i]]=fld[6]; - } - } - } -for (a in citilats) print a, citilats[a] > "citilats-dump" -for (a in citilngs) print a, citilngs[a] > "citilngs-dump" - while (getline goog < "geo_cache") { - split(goog,fld,"|"); - googlats[fld[1]] = fld[2]; - googlngs[fld[1]] = fld[3]; - } - print "read data files" > "/dev/stderr" - - start=1; - dfmt=" %11s: %d,\n"; - ffmt=" %11s: %.3f,\n"; - afmt=" %11s: \"%s\""; - afmt1=afmt ",\n"; - afmt2=afmt "\n }"; - printf "[" - printf "[" >> "teams.json" - } - /^ *\{/ { - for (i=1; i<=length(idx); i++) {atts[idx[i]]="";} - next; - } - /^ *\}/ { - city=toupper(atts["city"]); - sub(" *$","",city); - for (c in unicode) {gsub(c,unicode[c],city);} - country=ccode[atts["country"]]; - prov=toupper(atts["state_prov"]); - for (c in unicode) {gsub(c,unicode[c],prov);} - zip=atts["postal_code"]; - - # special fixes for Guam, zip weirdness, and some typoes - if (country=="") { - switch (zip) { - case /11073/: country="TW"; break; - case /34912/: country="TR"; break; - case /34469/: country="TR"; break; - case /93810/: country="IL"; break; - case /^[0-9]{4}$/: country="AU"; break; - case /^[0-9]{5}$/: - case /^[0-9]{5}-[0-9]{4}$/: country="US"; break; - case /^[0-9]{5}-[0-9]{3}$/: country="BR"; break; - case /^[A-Z][0-9][A-Z] [0-9][A-Z][0-9]$/: - country="CA"; break; - case /^[0-9]{7}$/: country="IL"; break; - } - } - if (country=="SE" && match(zip,/^[0-9]{5}/)) zip=substr(zip,1,3) " " substr(zip,4,2); - if (country=="US" && prov=="GUAM") country="GU"; - if (country=="US" && prov=="PUERTO RICO") country="PR"; - if (country=="CL" && prov=="REGION METROPOLITANA DE SANTIAGO") { - prov="SANTIAGO METROPOLITAN"; - } - if (country=="CN" && prov=="HUNAN") { - prov="HENAN"; - } - if (country=="GR" && prov=="THESSALIA") { - prov="THESSALY"; - } - if (country=="MX" && city=="SAN LUIS POTOTOSI") { - city="SAN LUIS POTOSI"; - } - if (country=="MX" && prov=="DISTRITO FEDERAL") { - prov="MEXICO CITY"; - } - if (country=="TR" && city=="CEKMEKOY") { - city="CEKMEKOEY"; - } - if (country=="US" && city=="NEW YORK") { - city="NEW YORK CITY"; - } - if (country=="US" && prov =="PA" && city=="WARMINSTER") { - city="WARMINSTER HEIGHTS"; - } - if (country=="US" && prov =="MO" && city=="LEES SUMMIT") { - city="LEE\047S SUMMIT"; - } - if (country=="NL" && prov =="NOORD-BRABANT") { - prov = "NORTH BRABANT"; - } - if (country=="DO" && prov =="SANTO DOMINGO" && city==prov) { - prov="NACIONAL"; - } - if (country=="IL") { - #if (prov=="HAMERKAZ (CENTRAL)") prov="CENTRAL DISTRICT"; - #if (prov=="HAZAFON (NORTHERN)") prov="NORTHERN DISTRICT"; - #if (prov=="HADAROM (SOUTHERN)") prov="SOUTHERN DISTRICT"; - #if (prov=="TEL-AVIV") prov="TEL AVIV"; - prov="IL" - } - if (country=="JP" && length(zip)==7) { - sub("...","&-",zip); - } - if (country == "CA") zip=substr(zip,1,3); # special for Canada - # end fixes - - lat=""; - lon=""; - if ((country "|" zip) in ziplats) { - atts["lat"]=ziplats[country "|" zip]; - atts["lng"]=ziplngs[country "|" zip]; - } else if ((country "|" prov "|" city) in citilats) { - atts["lat"]=citilats[country "|" prov "|" city]; - atts["lng"]=citilngs[country "|" prov "|" city]; - } else { - place=sprintf("%s, %s %s, %s",city, prov, zip, country); - if (place in googlats) { - atts["lat"]=googlats[place]; - atts["lng"]=googlngs[place]; -#print place "|" atts["lat"] "|" atts["lng"] "|" atts["team_number"] > "places2" - } else { - print place > "data/places" - print "Did not find team "atts["key"]" @ place", place > "/dev/stderr" - #print "You can find this with ./ask_google" > "/dev/stderr" - } - } - printf "%s\n {\"team_number\": %4d, \"lat\":%7.3f, \"lng\": %8.3f}", - start?"":",", atts["team_number"], - atts["lat"], atts["lng"] >> "teams.json" -#printf "%s.%s|%.3f|%.3f|%5d|\n", country, prov, atts["lat"], atts["lng"], atts["team_number"] > "provcheck" - printf "%s\n {\n", start?"":"," - printf dfmt, "team_number", atts["team_number"] - printf dfmt, "rookie_year", atts["rookie_year"] - printf ffmt, "lat", atts["lat"] - printf ffmt, "lng", atts["lng"] - printf afmt1, "website", atts["website"] - printf afmt1, "nickname", atts["nickname"] - printf afmt1, "motto", atts["motto"] - printf afmt2, "location", sprintf( \ - "%s, %s %s, %s", atts["city"], - atts["state_prov"], atts["postal_code"], - atts["country"]) - start=0; - next; - } - /^ *"/ {field=$2; - t=gensub("\", *$","",1); - sub(field,"",t); - sub("^[ :\",]*","",t); - sub("[ :\",]*$","",t); - sub(/\\$/,"&\"",t); - atts[field]=t; -#printf "%s|%s|\n",field,t > "attparser" - } - END { - printf "\n]\n" - printf "\n]\n" >> "teams.json" - }' data/merged ) > teamFullInfo.json diff --git a/geo_cache b/geo_cache index daf6eeb..b5510c1 100644 --- a/geo_cache +++ b/geo_cache @@ -1,25 +1,18 @@ -34, ISTANBUL 34034, TR|40.993|28.910 -ADAMA, OROMIYA None, ET|8.534|39.264 APO, HESSEN 09005, DE|50.042|8.329 ARARA NEGEV, IL 84911, IL|31.158|35.018 ASHER, OKLAHOMA 74862, US|34.986|-96.933 BAYRAMPASA, ISTANBUL 34030, TR|41.040|28.915 BET-HASMONAI, IL 99789, IL|31.888|34.905 BINYAMINA, IL 3050654, IL|32.526|34.937 -CHANGHUA, CHANGHUA 50055, TW|24.076|120.563 CHIAYI, CHIAYI 62153, TW|23.489|120.436 DALIAT EL CARMEL, IL 30056, IL|32.694|35.051 -EMEK HEFER, IL 40250, IL|32.333|34.890 EMEK HEFER, IL 4287500, IL|32.333|34.890 EVEN-YEHUDA, IL 40500, IL|32.268|34.881 FATIH, ISTANBUL 34100, TR|41.014|28.952 FOUSHAN, GUANGDONG 528000, CN|23.021|113.124 GADERA, IL 77400, IL|31.815|34.780 HAKFAR HAYAROK, IL 4780000, IL|32.133|34.826 -HERZELIA, IL 4673004, IL|32.173|34.828 HOD-HA'SHARON, IL 45101, IL|32.146|34.892 -HONG KONG, HONG KONG (XIANGGANG) 00000, CN|22.275|114.174 -JATT, IL 71799, IL|32.411|35.021 KADIMA-ZORAN, IL 6092000, IL|32.281|34.911 KARTAL, ISTANBUL 34863, TR|40.908|29.172 KEFAR BLUM, IL 12150, IL|33.172|35.607 @@ -28,144 +21,230 @@ KFAR HANOAR NEURIM, IL 4227822, IL|32.374|34.862 KFAR MANDA, IL 17907, IL|32.806|35.259 KIBUTZ E'IN SHEMER, IL 99999, IL|32.944|35.084 KOCAELI, KOCAELI 41490, TR|40.763|29.920 -MAAGAN MICHAEL, IL 37805, IL|32.541|34.913 -MACAU, MACAU (AOMEN) 999078, CN|22.190|113.536 MACAU, MACAU 999078, CN|22.190|113.536 -MAJD EL KURUM, IL 20190, IL|32.916|35.229 MEGIDDO REGIONAL COUNCIL, IL 99999, IL|32.580|35.171 -MISGAV, IL 20179, IL|32.859|35.260 MODI'IN-MACCABIM REU'T, IL 7179902, IL|31.887|35.019 NAVAJO MOUNTAIN, UTAH 00000, US|37.047|-110.799 -PARDES HANA, IL 000000, IL|32.475|34.964 PETACH TIKVAH, IL 99999, IL|32.091|34.887 PETAH TIQUA, IL 4938827, IL|32.091|34.817 PRAGUE, PRAHA 8 18000, CZ|50.109|14.475 RECOLETA, SANTIAGO METROPOLITAN 8420549, CL|-33.406|-70.634 RISHON LE TZION, IL 7549112, IL|31.969|34.781 ROSH HAYIN, IL 4805657, IL|32.094|34.974 -SANTO DOMINGO, DISTRITO NACIONAL (SANTO DOMINGO) 00000, DO|18.489|-69.942 SANTO DOMINGO, DISTRITO NACIONAL 00000, DO|18.489|-69.942 -SHENZHEN, SHANDONG 518000, CN|22.544|114.070 -SHOHAM, IL 60850, IL|31.997|34.950 TAYBE, IL 40400, IL|32.264|34.999 WURI DIST., TAICHUNG 41401, TW|24.090|120.634 -TEL SHEVA, IL 8495000, IL|31.248|34.841 HONG KONG, HONG KONG 00000, CN|22.276|114.228 HANOI, HA NOI 150000, VN|21.007|105.796 NAN CHONG, SICHUAN 637000, CN|30.800|106.082 MISGAV, IL 2017900, IL|32.858|35.256 -SINGAPORE, CENTRAL SINGAPORE 738547, SG|1.425|104.773 PARDES HANA, IL 37100, IL|32.475|34.943 -JADEH MAHBAS, HER\U0101T 3001, AF|34.334|62.158 -CONCORD, CALIFORNIA , |37.983|-122.039 -MARINA, CALIFORNIA , |36.718|-121.916 -CHULA VISTA, CALIFORNIA , |32.636|-117.061 -SAN FRANCISCO, CALIFORNIA , |37.758|-122.578 -RIVERSIDE, CALIFORNIA , |33.971|-117.368 -BURTON, MICHIGAN , |43.013|-83.594 -MOUNT PLEASANT, MICHIGAN , |43.603|-84.773 -BROOKLYN, MICHIGAN , |42.092|-84.286 -BREWSTER, NEW YORK , |41.442|-73.603 +JADEH MAHBAS, HERAT 3001, AF|34.334|62.158 TRIPOLI, TARABULUS 00218, LY|32.883|13.153 -ZHUQI, CHIAYI , TW|23.508|120.92 HATAY, HATAY 31100, TR|36.202|36.161 -WIXOM, MICHIGAN , |42.526|-83.603 JAFFA OF NAZARETH, IL 5902, IL|32.684|35.273 -LACHINE, QUEBEC h8s, CA|45.443|-73.681 -CHICAGO, ILLINOIS , |41.832|-88.294 -JAMESTOWN, NORTH DAKOTA , |46.910|-98.746 -TAIPEI, TAIPEI SPECIAL MUNICIPALITY , TW|25.018|121.493 -FRASER, MICHIGAN , |42.55|-82.947 -SCOTTVILLE, MICHIGAN , |43.957|-86.286 -PALMDALE, CALIFORNIA , |34.592|-118.176 -BRIDGEVIEW, ILLINOIS , |41.724|-87.803 -NEW TAIPEI CITY, TAIPEI , TW|25.015|121.464 -BESSEMER, MICHIGAN , |46.478|-90.085 CHANGHUA CITY, CHANGHUA , TW|24.078|120.566 -TAIPEI CITY, TAIPEI , TW|25.044|121.537 -MICHIGAN CENTER, MICHIGAN , |42.233|-84.391 -KALAMAZOO, MICHIGAN , |42.243|-85.573 -ANN ARBOR, MICHIGAN , |42.314|-83.675 -FRESNO, CALIFORNIA , |36.787|-119.881 -GRAND LEDGE, MICHIGAN , |42.746|-84.727 -NEW TAIPEI CITY, TAIPEI 242, TW|25.023|121.428 -SWANVILLE, MINNESOTA , |45.912|-94.662 -CIUDAD DE MEXICO, MEXICO CITY , MX|19.452|-99.203 -LAFAYETTE, CALIFORNIA , |37.874|-122.133 -MUSKEGON, MICHIGAN , |43.237|-86.327 -SOUTHFIELD, MICHIGAN , |42.454|-83.218 -BAN QIAO DISTRICT, TAIPEI SPECIAL MUNICIPALITY , TW|25.009|121.453 -FELCH, MICHIGAN , |46.011|-87.960 -DETROIT, MICHIGAN , |42.352|-83.380 -DEPOK, JAWA BARAT , ID|-6.391|106.804 -THIELLS, NEW YORK , |41.215|-74.015 -\U00DCSKUDAR, ISTANBUL , TR|41.035|28.907 -WEST BRANCH, MICHIGAN , |44.263|-84.215 -IRVINGTON, NEW YORK , |41.035|-73.898 -WARREN, MICHIGAN , |42.515|-82.975 -IRON RIVER, MICHIGAN , |46.082|-88.634 -ROYAL OAK, MICHIGAN , |42.537|-83.18 -MONTROSE, MICHIGAN , |43.181|-83.887 -GREENFIELD, CALIFORNIA , |36.324|-121.261 -BAKERSFIELD, CALIFORNIA , |35.324|-119.014 -COLYTON, NEW SOUTH WALES , AU|-33.782|150.795 -ROCK, MICHIGAN , |45.986|-87.111 -ORCHARD LAKE, MICHIGAN , |42.593|-83.357 -VASSAR, MICHIGAN , |43.376|-83.585 -SHENANDOAH, IOWA , |40.751|-95.361 -INKSTER, MICHIGAN , |42.304|-83.334 -HARBOR SPRINGS, MICHIGAN , |45.434|-84.988 -CLINTON TOWNSHIP, MICHIGAN , |42.462|-83.020 -MARION, MICHIGAN , |44.102|-85.116 -DAYTON, NEVADA , |39.239|-119.572 -GARDEN CITY, MICHIGAN , |42.336|-83.328 -KALKASKA, MICHIGAN , |44.737|-85.1895606 -BANGOR, MICHIGAN , |42.307|-86.123 -BATTLE CREEK, MICHIGAN , |42.321|-85.179 -IONIA, MICHIGAN , |42.981|-85.043 -NEW HAVEN, MICHIGAN , |42.725|-82.788 -THREE OAKS, MICHIGAN , |41.841|-86.618 -ISHPEMING, MICHIGAN , |46.488|-87.701 -LANSING, MICHIGAN , |42.739|-84.529 -WILLIAMSTON, MICHIGAN , |42.698|-84.276 -ROGERS CITY, MICHIGAN , |45.417|-83.826 -MARQUETTE, MICHIGAN , |46.550|-87.434 -CRYSTAL FALLS, MICHIGAN , |46.096|-88.339 -KADIKOY, ISTANBUL , TR|40.983|29.082 -BENTLEY, ALBERTA , CA|52.467|-114.049 -SAINT PAUL, MINNESOTA , |44.940|-93.246 -COON RAPIDS, MINNESOTA , |45.166|-93.320 -KYIV, KY\U00EFV , UA|50.402|30.253 -MOUNT CLEMENS, MICHIGAN , |42.596|-82.883 -MESICK, MICHIGAN , |44.406|-85.731 -OAK PARK, MICHIGAN , |42.461|-83.185 -OLIVIA, MINNESOTA , |44.770|-94.989 -ROMULUS, MICHIGAN , |42.167|-83.323 -ALBA, MICHIGAN , |44.977|-84.971 -RENO, NEVADA , |39.555|-119.797 -MEDFORD, MINNESOTA , |44.172|-93.250 -FLINT, MICHIGAN , |43.073|-83.742 -BELDING, MICHIGAN , |43.093|-85.223 -NOVI, MICHIGAN , |42.469|-83.440 -BESIKTAS, ISTANBUL , TR|41.045|29.012 -CORPUS CHRISTI, TX , |27.762|-97.440 +NEW TAIPEI CITY, CHANGHUA 242, TW|25.023|121.428 ZHUQI, CHIAYI 604, TW|23.502|120.529 -TAIPEI, TAIPEI SPECIAL MUNICIPALITY 11452, TW|25.018|121.226 -NEW TAIPEI CITY, TAIPEI 23159, TW|25.018|121.226 +NEW TAIPEI CITY, CHANGHUA 23159, TW|25.018|121.226 TAIPEI CITY, TAIPEI 234, TW|25.008|121.497 -BAN QIAO DISTRICT, TAIPEI SPECIAL MUNICIPALITY 22050, TW|25.012|121.457 +BAN QIAO DISTRICT, TAIPEI 22050, TW|25.012|121.457 DEPOK, JAWA BARAT 16451, ID|-6.383|106.823 TRIPOLI, TARABULUS NONE, LY|32.883|13.153 KANGSHAN, KAOHSIUNG 820, TW|22.812|120.332 EMEK YIZRAEL, IL 3659000, IL|32.654|35.155 DOULIU CITY, YUNLIN 640, TW|23.710|120.515 -TAINAN CITY, TAINAN MUNICIPALITY 741, TW|23.141|120.266 -KYIV, KY\U00EFV 03506, UA|50.402|29.972 +TAINAN CITY, TAINAN 741, TW|23.141|120.266 +KYIV, KYIV 03506, UA|50.402|29.972 WATERFORD, SAINT MICHAEL BB, BB|13.120|-59.605 -SOFIA, SOFIA-GRAD 1172, BG|42.696|23.184 -MERKEZ, AFYONKARAHISAR 03020, TR|38.757|30.485 BEER YAKOV, IL 7030861, IL|31.948|34.805 CABA, CIUDAD AUTONOMA DE BUENOS AIRES 1184, AR|-34.651|-58.391 -YARKA, IL 2496700, IL|32.960|35.176 -KYIV, KY\U00EFV 02000, UA|50.402|30.253 -, 050000, |43.217|76.664 +KAOHSIUNG, CHANGHUA 80748, TW|22.619|120.319 +TAINAN, CHANGHUA 700, TW|23.000|120.227 +TAINAN, CHANGHUA 70841, TW|23.000|120.227 +TAICHUNG, CHANGHUA 40864, TW|24.148|120.674 +SPRINGPORT, MICHIGAN , US|42.377|-84.703 +WOODBINE, MARYLAND , US|39.374|-77.057 +ERIE, MICHIGAN , US|41.794|-83.497 +NORTH YORK, ONTARIO , CA|43.756|-79.434 +, BEIJING , CN|40.146|116.384 +MEDFORD, NEW JERSEY , US|39.8607147|-74.8788073 +PETACH TIKWA, IL 99999, IL|32.0908579|34.852166 +RICHLAND, MICHIGAN , US|42.376216|-85.4639967 +BET-HASHMONAI, IL 99789, IL|31.890839|34.9103337 +MUNTENDAM, GRONINGEN 9747AS, NL|53.2402584|6.5307698 +PRAGUE, PRAHA, HLAVNI MESTO 18000, CZ|50.1095798|14.451279 +HADASSAH NEURIM YOUTH VILLAGE, IL 4227822, IL|32.3738466|34.8572044 +WURI, TAICHUNG 41401, TW|24.0823699|120.5524037 +BESIKTAS, ISTANBUL 34000, TR|41.0429611|29.0014496 +KONAK, IZMIR 35000, TR|38.4097671|27.1218416 +NEW TAIPEI CITY, NEW TAIPEI 23159, TW|25.0175862|121.4362949 +DAYA DIST., TAICHUNG 428, TW|24.1852333|120.5997002 +NEW TAIPEI CITY, NEW TAIPEI 242, TW|25.045346|121.4464831 +TAIPEI CITY, TAIPEI , TW|25.0409511|121.4964195 +, NEW TAIPEI , TW|25.0159056|121.4622138 +TOYOSU, TOKYO , JP|35.6520586|139.7712756 +DUNBARTON, NEW HAMPSHIRE , US|43.107332|-71.6598826 +SAINT LOUIS, MISSOURI , US|38.6532851|-90.3835475 +ZHONGHE DIST., NEW TAIPEI , TW|24.9903618|121.4629826 +IJAMSVILLE, MARYLAND , US|39.3365133|-77.3744387 +ST.ROSE, LOUISIANA , US|29.9567185|-90.3268771 +FUQUAY VARINA, NORTH CAROLINA , US|35.6110583|-78.8352049 +UNYE, ORDU , TR|41.1141796|37.2376452 +DURANGO, DURANGO , MX|24.0228392|-104.7177655 +TURKIYE, MERSIN , TR|36.7021097|33.2878613 +THORNHILL, ONTARIO , CA|43.8170848|-79.4532345 +PACKWOOD, IOWA , US|41.126295|-92.1027216 +SAINT JOHNS, FLORIDA , US|30.0770189|-81.5501501 +TAINAN CITY, TAINAN , TW|23.1229948|120.1312971 +KONAK, IZMIR , TR|38.4219611|27.0941414 +KOCAELI, KOCAELI , TR|40.7711737|29.8994815 +NEW TAIPEI CITY, NEW TAIPEI , TW|24.9875278|121.3645928 +JACKSON JUNCTION, IOWA , US|43.1007975|-92.0648126 +MONTROSE, IOWA , US|40.526609|-91.4249408 +SCARBOROUGH, ONTARIO , CA|43.7634634|-79.3640716 +KARTAL, ISTANBUL , TR|40.920666|29.1617245 +EWA BEACH, HAWAII , US|21.318183|-158.0267996 +ASHBY, MINNESOTA , US|46.093153|-95.8205938 +DEL VALLE, TEXAS , US|30.2152693|-97.6640032 +BARRETT, MINNESOTA , US|45.9160104|-95.9056736 +BEYOGLU, ISTANBUL , TR|41.0431074|28.93236 +MARION CENTER, PENNSYLVANIA , US|40.7707364|-79.0558263 +KOCAELI, KOCAELI , TR|40.7711737|29.8994815 +EYUP, ISTANBUL , TR|41.0543315|28.9330318 +BAKI, BAKI , AZ|40.3947695|49.7148743 +COEUR D ALENE, IDAHO , US|47.7014946|-116.8621144 +KENTS HILL, MAINE , US|44.4030425|-70.010878 +QUERETARO, QUERETARO , MX|20.6122835|-100.4802579 +KONAK, IZMIR , TR|38.4219611|27.0941414 +KADIKOY, ISTANBUL , TR|40.9811925|29.0280335 +NEW TAIPEI, NEW TAIPEI , TW|25.0174843|121.4632758 +PLEASANT VALLEY, IOWA , US|41.5652511|-90.5394227 +SOFIA, SOFIA , BG|42.6955991|23.1838617 +WYNNEWOOD, PENNSYLVANIA , US|39.9996434|-75.3070613 +SAINT-CHRISTOL-LES-ALES, GARD , FR|44.0830399|4.0331595 +SAO JOAO DEL-REI, MINAS GERAIS , BR|-21.1159616|-44.2756162 +NILUFER, BURSA , TR|40.1755304|28.6893652 +KAOHSIUNG CITY, KAOHSIUNG , TW|22.6423868|120.3273035 +PANACA, NEVADA , US|22.628046|120.2916941 +KRAKOW, MALOPOLSKIE , PL|50.0469018|19.8647892 +AIEA, HAWAII , US|21.3858601|-157.9398791 +SINGAPORE, NORTH WEST 11152, SG|1.352722|103.8050945 +BAKIRKOY, ISTANBUL , TR|40.9808845|28.7998655 +PARANGABA, CEARA , BR|-3.7777775|-38.5742262 +NEW BOSTON, MICHIGAN , US|42.1627331|-83.4028668 +TARABYA, ISTANBUL, ISTANBUL , TR|41.1396291|29.04259 +EWEN, MICHIGAN , US|46.5352439|-89.2814486 +SAO JOSE OPERARIO, AMAZONAS , BR|-3.0667725|-59.953113 +TEKONSHA, MICHIGAN , US|42.0957724|-84.990029 +JACAREPAGUA, RIO DE JANEIRO , BR|-22.9625371|-43.386836 +VALBONNE, ALPES-MARITIMES , FR|43.6320098|7.0272936 +ALDIE, VIRGINIA , US|38.9732736|-77.6379941 +ROSEDALE, LOUISIANA , US|30.4453785|-91.4664732 +MIDDLETON, MICHIGAN , US|43.1834725|-84.7088722 +CHASSELL, MICHIGAN , US|47.0283336|-88.5252192 +WARSZAWA, MAZOWIECKIE , PL|52.2479393|21.0180544 +ARDEN, NORTH CAROLINA , US|35.4656488|-82.5178624 +GIVAT SHMUEL, IL , IL|32.0772218|34.8503353 +SDOT NEGEV, IL , IL|31.4215619|34.5482821 +ROSEMARY BEACH, FLORIDA , US|30.2824134|-86.0186193 +ZHONGHE DIST, NEW TAIPEI , TW|24.9962178|121.4853103 +131 ARGYLE ST. KLN, HONG KONG , CN|22.321303|114.173225 +LYNDON CENTER, VERMONT , US|44.5416086|-72.0153219 +MATE ASHER, IL , IL|33.0059802|35.148879 +FIFE LAKE, MICHIGAN , US|44.5769496|-85.3506136 +LUZERNA, SANTA CATARINA , BR|-27.1291581|-51.4683177 +BEYLIKDUZU, ISTANBUL , TR|40.9910381|28.6498144 +HENRICO, VIRGINIA , US|37.6307265|-77.5439961 +ATHENS, ATTIKI , GR|37.9838096|23.7275388 +HASTINGS-ON-HUDSON,, NEW YORK , US|40.994542|-73.8787461 +NOVA MUTUM, MATO GROSSO , BR|-13.8169976|-56.0884274 +CACERES, MATO GROSSO , BR|-16.0873856|-57.6754995 +SORRISO, MATO GROSSO , BR|-12.5407237|-55.7226752 +GOSNELLS, 6110, WESTERN AUSTRALIA , AU|-32.0824185|115.990413 +SARDES, MANISA , TR|38.4794543|28.0311938 +HATAY, HATAY , TR|36.2042794|36.1618938 +DORCHESTER, SOUTH CAROLINA , US|33.1680383|-80.543845 +FOXBORO, MASSACHUSETTS , US|42.0653812|-71.2478251 +GAZIOSMANPASA, ISTANBUL , TR|41.0759477|28.9004553 +BRONX, NEW YORK , US|40.8447819|-73.8648268 +MOCA, ESPAILLAT , DO|19.3908634|-70.5231108 +CENTERVILLE, LOUISIANA , US|29.7597872|-91.428367 +ANN ARBOR, CITY OF, MICHIGAN , US|42.2808256|-83.7430378 +GENESEE, MICHIGAN , US|43.0777289|-83.6773928 +FLANDERS, NEW JERSEY , US|40.8418677|-74.7161489 +EIN SHEMER, IL 99999, IL|32.461917|35.00673 + , IL 40500, IL|40.6331249|-89.3985283 +BEIT HASHMONAY, IL 99789, IL|31.890298|34.917599 +MINXIONG TOWNSHIP, CHIAYI , TW|23.5202655|120.4403808 +APODACA NUEVO LEON, NUEVO LEON 66451, MX|25.776468|-100.1858743 +PARDES HANA, IL 4227822, IL|32.4728028|34.9742001 +WELLESLEY HILLS, MASSACHUSETTS , US|42.3084301|-71.2786677 +CHIAYI, CHIAYI , TW|23.4797312|120.449629 +SANCHONG ,DIST., NEW TAIPEI , TW|25.0614534|121.4867114 +SHIBUYA-KU, TOKYO , JP|35.6619707|139.703795 +TAOYUAN CITY, TAOYUAN , TW|24.9936281|121.3009798 +SAFRANBOLU, KARABUK , TR|41.249306|32.683128 +DIYARBAKIR, DIYARBAKIR , TR|37.9249733|40.2109826 +IZMIR, IZMIR , TR|38.423734|27.142826 +MENEMEN, IZMIR , TR|38.613023|27.0323305 +BELIZE CITY, BELIZE , BZ|17.5045661|-88.1962133 +HONG KONG, HONG KONG , CN|22.3193039|114.1693611 +SHULIN, NEW TAIPEI , TW|24.9815605|121.4198606 +BAYRAMPASA, ISTANBUL , TR|41.0481503|28.9004553 +SINCAN, ANKARA , TR|39.8544476|32.424074 +LIVINGSTON COUNTY, MICHIGAN , US|42.6207951|-83.8473015 +AKCABURGAZ /ESENYURT, ISTANBUL , TR|41.0654136|28.6323132 +TAGUATINGA, DISTRITO FEDERAL , BR|-15.8335644|-48.057109 +LINHAS, ESPIRITO SANTO , BR|-19.3990168|-40.0653465 +MYOZAI-GUN,KAMIYAMA-CHO, TOKUSHIMA , JP|33.9672275|134.3504705 +KUCUKCEKMECE, ISTANBUL , TR|41.0212376|28.7780987 +BUNKYO-KU, TOKYO , JP|35.7078686|139.7524369 +DIYARBAKIR, DIYARBAKIR , TR|37.9249733|40.2109826 +KINGWOOD, TEXAS , US|30.0500575|-95.1846057 +SARIYER, ISTANBUL , TR|41.1814742|29.0385466 +BARK RIVER, MICHIGAN , US|45.7102456|-87.3048536 +COVERT, MICHIGAN , US|42.2745346|-86.2767995 +USTI NAD LABEM, USTI NAD LABEM , CZ|50.6611164|14.0531455 +KEMALPASA, IZMIR , TR|38.4275|27.4188 +ELKTON, OREGON , US|43.6376174|-123.568152 +FATIH, ISTANBUL , TR|41.0168639|28.9470422 +WINSTON SALEM, NORTH CAROLINA , US|36.0998596|-80.244216 +FATIH, ISTANBUL , TR|41.0168639|28.9470422 +MONTEZUMA, NEW MEXICO , US|35.6521842|-105.2766154 +COMOX, BRITISH COLUMBIA , CA|49.6735133|-124.9282659 +MANILA, NATIONAL CAPITAL REGION , PH|14.5995124|120.9842195 +SAINT PETERSBURG, FLORIDA , US|27.7671271|-82.6384451 +GRAND MARAIS, MICHIGAN , US|46.671789|-85.9841234 +WEST ROXBURY, MASSACHUSETTS , US|42.2782373|-71.1599757 +MODI'IN-MACCABIM-RE'UT, IL , IL|31.890267|35.010397 +WALKERVILLE, MICHIGAN , US|43.7144551|-86.1245131 +VESTABURG, MICHIGAN , US|43.3989671|-84.9052528 +WINNETKA, CALIFORNIA , US|34.208337|-118.5710684 +BAKER, FLORIDA , US|30.7971318|-86.681344 +UNIONVILLE, ONTARIO , CA|43.8655329|-79.3106862 +DONGSHAN, YILAN , TW|24.631919|121.7537404 +KARABUK, KARABUK , TR|41.19562|32.622654 +KOCAELI/GEBZE, KOCAELI , TR|40.8025157|29.4397941 +NORTH CLARENDON, VERMONT , US|43.5655235|-72.9661208 +LAWRENCE TOWNSHIP, NEW JERSEY , US|40.2727829|-74.7348792 +MINOT AFB, NORTH DAKOTA , US|48.4157509|-101.3386723 +FATIH, ISTANBUL , TR|41.0168639|28.9470422 +AVCILAR, ISTANBUL , TR|41.0153479|28.7314618 +TAGUATINGA, DISTRITO FEDERAL , BR|-15.8335644|-48.057109 +BURSA, BURSA , TR|40.1885281|29.0609636 +COOKS, MICHIGAN , US|45.9177507|-86.4762568 +WARSAW, MAZOWIECKIE , PL|52.2296756|21.0122287 +ENGADINE, MICHIGAN , US|46.1170049|-85.5712686 +TILLSONBURG, ONTARIO , CA|42.8658879|-80.7333175 +BRAITHWAITE, LOUISIANA , US|29.8728375|-89.9650134 +BESIKTAS, ISTANBUL , TR|41.044864|29.018614 +PALMIRA VALLE DEL CAUCA, VALLE DEL CAUCA , CO|3.5378587|-76.297237 +SDE ELIEZER, IL , IL|33.04647|35.56525 +DAMASCUS, ?ALAB , SY|33.5123768|36.2960875 +GIRNE, LEFKOSIA , CY|35.3322825|33.3195479 +KAYAPINAR, DIYARBAKIR , TR|37.9653913|40.0815606 +KIGALI, VILLE DE KIGALI , RW|-1.9440727|30.0618851 +PALMIRA, VALLE DEL CAUCA , CO|3.5378587|-76.297237 +VANDERBILT, PENNSYLVANIA , US|40.0331301|-79.6614284 +YEREVAN, EREVAN , AM|40.1872023|44.515209 \ No newline at end of file diff --git a/get_all_data b/get_all_data deleted file mode 100755 index e0613f1..0000000 --- a/get_all_data +++ /dev/null @@ -1,10 +0,0 @@ -#!/bin/bash -# get_all_data -mkdir data -./get_postal # get zip/postal code/city lookup info -./get_events # get competition data from theblualliance -./get_lists # get team data from thebluealliance -./merge_lists # merges team data from thebluealliance into one file -./build_teamInfo # geocodes and builds team numbers, lat, lon, etc. - -#-rwxr-xr-x 1 Daddy None 406 Oct 6 20:23 ask_google diff --git a/get_events b/get_events deleted file mode 100755 index 27456c5..0000000 --- a/get_events +++ /dev/null @@ -1,50 +0,0 @@ -#!/bin/bash -f -#get_events -api="https://www.thebluealliance.com/api/v3/events/" -auth="?X-TBA-Auth-Key=$(cat TBA-auth)" -wget -Odata/events-full "$api$(cat year)$auth" -(echo "// Data extracted from thebluealliance.com at $(TZ=utc date -Iseconds)" - echo "// Built by scraper at https://github.com/FIRSTMap/FIRSTMap-scraper" - echo - gawk -F: ' BEGIN {first=1; sn=1 - printf "var events = [" - } - /"event_type"/ {event_type[sn]=gensub("\"*, *$","",1,$2); - gsub("^ *\"*","",event_type[sn]);} - /"short_name"/ {short_name[sn]=gensub("\"*, *$","",1,$2); - gsub("^ *\"*","",short_name[sn]);} - /"name"/ {name[sn]=gensub("\"*, *$","",1,$2); - gsub("^ *\"*","",name[sn]);} - /"key"/ {key[sn]=gensub("\"*, *$","",1,$2); - gsub("^ *\"*","",key[sn]);} - /"lat"/ {lat[sn]=gensub("\"*, *$","",1,$2); - gsub("^ *\"*","",lat[sn]);} - /"lng"/ {lng[sn]=gensub("\"*, *$","",1,$2); - gsub("^ *\"*","",lng[sn]);} - /"parent_event_key"/ {pek=gensub("\"*, *$","",1,$2); - gsub("^ *\"*","",pek);} - /^ }/ { - if (pek == "null") {sn++;} } - END { - for (i=1; i0) {lat[i]-=0.0001; lng[i]+=0.0001;} - } - for (i=1; i98)?"offseason":"district"), - (length(short_name[i])>0) ? short_name[i]: name[i] - first=0; - } - printf "\n]\n" - }' data/events-full ) > events.js diff --git a/get_lists b/get_lists deleted file mode 100755 index 4a95cca..0000000 --- a/get_lists +++ /dev/null @@ -1,19 +0,0 @@ -#!/bin/bash -# get_lists -api="https://www.thebluealliance.com/api/v3/teams/" -auth="?X-TBA-Auth-Key=$(cat TBA-auth)" -year=$(cat YEAR) -i=0; -go=1 -TZ=utc date -Iseconds > data/team-time -while /bin/true -do wget -O"data/teams.$i" "$api$year/$i$auth" - go=$(grep -c -s key "data/teams.$i") - if [ $go -ge 1 ] - then echo "got data/teams.$i" - else echo "empty file for data/teams.$i; terminating" - rm "data/teams.$i" - exit 0 - fi - i=$(expr $i + 1) -done diff --git a/get_postal b/get_postal deleted file mode 100755 index 4ec0245..0000000 --- a/get_postal +++ /dev/null @@ -1,15 +0,0 @@ -#!/bin/bash -wget -O data/allCountries.zip \ - http://download.geonames.org/export/zip/allCountries.zip -zcat data/allCountries.zip > data/allCountries -wget -O data/allCountries.readme \ - http://download.geonames.org/export/dump/readme.txt -wget -O data/cities1000.zip \ - http://download.geonames.org/export/dump/cities1000.zip -zcat data/cities1000.zip > data/cities1000 -wget -O data/cities1000.readme \ - http://download.geonames.org/export/dump/readme.txt -wget -O data/admin1CodesASCII.txt \ - http://download.geonames.org/export/dump/admin1CodesASCII.txt -wget -Odata/countryInfo.txt \ - http://download.geonames.org/export/dump/countryInfo.txt diff --git a/merge_lists b/merge_lists deleted file mode 100755 index 57dc7c5..0000000 --- a/merge_lists +++ /dev/null @@ -1,18 +0,0 @@ -#!/bin/bash -#merge_lists -# merges the 50-team data files into a single file, and slims down the -# home_championship attribute to an atomic value. -year=$(cat YEAR) -year=2018 -files=$(echo data/teams.? data/teams.??) -echo $files -awk -F'\0' 'BEGIN {start=1; wai=0} - /[\[\]]/ {next} - /^ *}/ {next} - /^ *{/ {if (start) {start=0} else {print " },"; }; } - /home_championship/ {if (!/null/) wai=1; } - /'$year'.:/ {sub(" .'$year'","\"home_championship"); sub("$",","); wai=0} - {if ( wai == 1 ) {next}} - {gsub("null","\"\"");} - {print} - END {printf " }\n";}' $files > data/merged diff --git a/scraper.py b/scraper.py new file mode 100755 index 0000000..8576790 --- /dev/null +++ b/scraper.py @@ -0,0 +1,648 @@ +#!/usr/bin/env python3 +""" +Licensed under the MIT License. +See LICENSE file for more information. +""" + +import json +import re +import sys +import unicodedata +from pathlib import Path +from zipfile import ZipFile +import argparse +from datetime import datetime + +import requests +import tbapy + +# +# Define constants that the program uses +# + +# Load the auth key +AUTH_PATH = Path('tba_token.txt') +if not AUTH_PATH.exists(): + print('Error: the tba_token.txt file does not exist! You must generate a' + + ' Read API authorization key on The Blue Alliance website. This can be' + + ' done at: https://www.thebluealliance.com/account. Place the generated' + + ' Read API Key in a file named tba_token.txt') + sys.exit() + +AUTH_KEY = AUTH_PATH.read_text().strip() + +# The directory where downloaded GeoNames data is cached +CACHE_DIR = Path.cwd() / 'cache' + +# When the latitude/longitude location of a place cannot be found, it is put +# into the broken_places file. This file is deleted (and recreated if needed) +# with each run of the scraper. +BROKEN_PLACES_FILE = CACHE_DIR / 'broken_places' + + +class GeoNamesFile(): + """ + A helper class to hold the url and name of a GeoNames file as + well as if it needs to be unzipped after download. + """ + + def __init__(self, url, name, unzip): + self.url = url + self.name = name + self.unzip = unzip + + +# What GeoNames files to download, name to save them as (within +# CACHE_DIR), and whether or not they need to be unzipped. +POSTAL_FILES = [ + GeoNamesFile(url='https://download.geonames.org/export/zip/allCountries.zip', + name='allCountries.zip', + unzip=True), + GeoNamesFile(url='https://download.geonames.org/export/dump/readme.txt', + name='allCountries.readme', + unzip=False), + GeoNamesFile(url='https://download.geonames.org/export/dump/cities1000.zip', + name='cities1000.zip', + unzip=True), + GeoNamesFile(url='https://download.geonames.org/export/dump/cities500.zip', + name='cities500.zip', + unzip=True), + GeoNamesFile(url='https://download.geonames.org/export/dump/readme.txt', + name='cities1000.readme', + unzip=False), + GeoNamesFile(url='https://download.geonames.org/export/dump/admin1CodesASCII.txt', + name='admin1CodesASCII.txt', + unzip=False), + GeoNamesFile(url='https://download.geonames.org/export/dump/countryInfo.txt', + name='countryInfo.txt', + unzip=False) +] + +# The attributes to copy from teams into the output. In the original AWK +# scripts, this was automatically loaded from a file called `attribs`. These +# are the attributes that are stored in `teamFullInfo.json`. `team.json` only +# has the team numbers and their associated latitude and longitude +# coordinates. +TEAM_ATTRIBS = [ + 'address', + 'city', + 'country', + 'gmaps_place_id', + 'gmaps_url', + 'home_championship', + 'key', + 'lat', + 'lng', + 'location_name', + 'motto', + 'name', + 'nickname', + 'postal_code', + 'rookie_year', + 'state_prov', + 'team_number', + 'website' +] + +# Additional country code mappings. Currently, this adds country +# codes for Czech Republic (CZ) and Chinese Taipei (TW) because +# those are the names used by FIRST/TBA, but Geonames has those +# countries listed as Czechia and Taiwan, respectively. +# It also adds the mapping of USA to US because TBA returns USA +# but Geonames has the country name United States. +# This dictionary is used in load_geonames_data +EXTRA_COUNTRY_CODES = { + 'Chinese Taipei': 'TW', + 'Czech Republic': 'CZ', + 'USA': 'US', + 'Türkiye': 'TR', + 'Netherlands': 'NL' +} + +# The chunk size used when downloading files from GeoNames. +DOWNLOAD_CHUNK_SIZE = 16384 + +# +# Initial program setup +# + +# Set up caching +if not CACHE_DIR.exists(): + CACHE_DIR.mkdir() +elif CACHE_DIR.is_file(): + print(f'Error: file "{str(CACHE_DIR)}" exists where the cache directory' + + ' is supposed to be created! Please delete the file!') + sys.exit() + +# Delete the current broken_places file if it exists +if BROKEN_PLACES_FILE.exists(): + BROKEN_PLACES_FILE.unlink() + + +# +# Function definitions +# + +def get_geonames_data(use_cache): + """ + Download and unzips all the information from GeoNames into + CACHE_DIR. + """ + print('Downloading GeoNames data...') + + for file in POSTAL_FILES: + print(f'Downloading {file.url}...') + + path = CACHE_DIR / file.name + + # Download the file from the URL and save it (except in cache mode, + # where the already-downloaded file is used if it exists). Download in + # streaming mode so the entire file is not loaded into memory + # before saving. + if not use_cache or not path.exists(): + with requests.get(file.url, stream=True) as req: + # If there is an error downloading, just crash + req.raise_for_status() + + with open(path, 'wb') as writer: + for chunk in req.iter_content(chunk_size=DOWNLOAD_CHUNK_SIZE): + writer.write(chunk) + + # If the file is a zip file, extract it to the data directory + if file.unzip: + print('Unzipping...') + with ZipFile(path, 'r') as zip: + zip.extractall(CACHE_DIR) + + +def load_geonames_data(): + """Load all the GeoNames data from the downloaded files.""" + print('Loading GeoNames data...') + + # + # Definitions + # + geo_names = {} + + # Function to read this specific format of TSV files GeoNames provides. It + # reads the file line by line (row by row) and breaks each row into + # columns, calling callback with each row (as a list of columns). The sep + # parameter changes the separator, so it can read CSV files, etc. as well. + def read_tsv(file, callback, sep='\t'): + with open(file, 'rt', encoding='utf-8') as reader: + for line in reader: + # Skip empty and blank lines and comments (which start with #) + if line.lstrip().startswith('#') or not line or line.isspace(): + continue + row = line.split(sep) + callback(row) + + # + # Initialize the country codes table + # + print('Loading country code mappings...') + geo_names['ccodes'] = {} + + # Fill in the country codes table with data from CACHE_DIR/countryInfo.txt + def process_ccode_row(row): + geo_names['ccodes'][row[4]] = row[0] + + read_tsv(CACHE_DIR / 'countryInfo.txt', process_ccode_row) + + # Add/replace additional country code mappings + for country in EXTRA_COUNTRY_CODES: + geo_names['ccodes'][country] = EXTRA_COUNTRY_CODES[country] + + # Remove the empty string key from the table, if it exists + geo_names['ccodes'].pop('', None) + + # + # Initialize the zipLocs dictionary, which contains latitude and + # longitude coordinates for every zip code of every country in + # allCountries.txt. + # + print('Loading zip code locations...') + geo_names['zipLocs'] = {} + + # Fill in the zipLocs table with data from CACHE_DIR/allCountries.txt + # The first key is the country code (row[0]), the second key is the zip + # (row[1]), and row[9] and row[10] are latitude and longitude coordinates, + # respectively. + def process_zip_data_col(row): + # Name the country and zip all uppercase (some zip codes have letters) + ccode = row[0].upper() + zip = row[1].upper() + + # If the country's entry doesn't exist, initialize it. + if not ccode in geo_names['zipLocs']: + geo_names['zipLocs'][ccode] = {} + + # Assign the zip's lat and lng in the dictionary. + geo_names['zipLocs'][ccode][zip] = { + 'lat': row[9], + 'lng': row[10] + } + + read_tsv(CACHE_DIR / 'allCountries.txt', process_zip_data_col) + + # + # Initialize the adms dictionary, which maps administrative division + # codes to their ASCII encoded English names. For example, the + # administrative division code US.AK maps to the name Alaska. These + # mappings come from admin1CodesASCII.txt + # + print('Loading administrative division names...') + geo_names['adms'] = {} + + def process_admin_codes(row): + # row[0] is the administrative division code, row[2] is the ASCII + # encoded English name of the administrative division code. For + # example, the ascii encoded name for São Paulo would be Sao Paulo (ã + # is replaced with a) + geo_names['adms'][row[0]] = row[2].upper() + + read_tsv(CACHE_DIR / 'admin1CodesASCII.txt', process_admin_codes) + + # + # Initialize the cities dictionary, which holds the latitude and + # longitude coordinates for each city in cities1000.txt. The dictionary + # is actually a dictionary with country names as the keys, where each key + # maps to a dictionary with state/province names as keys, where each key + # maps to a dictionary with cities as keys, where each key maps to a + # dictionary with a lat and lng entry for the latitude and longitude of + # that city. + # + # In other words, the format is: + # + # geoNames['cities'] = { + # 'United States': { + # 'New Hampshire': { + # 'Manchester': { + # 'lat': 42.99564, + # 'lng': -71.45479 + # } + # # ... more cities in New Hampshire + # } + # # ... more states in the US + # } + # # ... more countries + # } + # + print('Loading city locations...') + geo_names['cities'] = {} + + # Load the latitude and longitude for each city in cities1000.txt and put + # them in geoNames['cities'] + def process_cities(row): + city_name_ascii = row[2].upper() + country_code = row[8] + state_code = row[10] + # Name of the administrative division (state, province, etc.) + # See comment above about the adms dictionary for further explanation. + admin_name_ascii = geo_names['adms'].get(f'{country_code}.{state_code}') + + # Ignore nonexistant administrative divisions + if not admin_name_ascii: + return + + # This helper function puts the latitude and longitude of the city + # into the cities table (see comment at the beginning of this section + # for information about the format of the cities table) + def setLatLng(country, state, city): + if not country in geo_names['cities']: + geo_names['cities'][country] = {} + + if not state in geo_names['cities'][country]: + geo_names['cities'][country][state] = {} + + geo_names['cities'][country][state][city] = { + 'lat': row[4], + 'lng': row[5] + } + + # Set the latitude and longitude of the city for the current row + setLatLng(country_code, admin_name_ascii, city_name_ascii) + + if country_code == 'TW': + # For Taiwan, use the city name for both the city name and + # administrative division name (this is how the locations come from + # TBA) + setLatLng(country_code, city_name_ascii, city_name_ascii) + elif country_code == 'IL': + # Apparently Israel has bad district names and sometimes uses the + # alternative city names, so the country code is used for the + # administrative division name and all alternative names for cities + # are added (in addition to the regular city name, which was + # already added above, before the if statement). Alternative names + # are put in all caps. + alt_names = row[3].upper().split(',') + + for name in alt_names: + setLatLng(country_code, country_code, name) + + read_tsv(CACHE_DIR / 'cities500.txt', process_cities) + + # + # Load the coordinates from the geo_cache file that were + # manually obtained. Some team locations have to be manually + # obtained because their locations cannot be resolved using + # the GeoNames data alone, often due to incomplete data from + # TBA. This usually involves the user of the scraper finding + # the actual location of the team by looking at information + # on TBA, etc., and then finding the lat/lng coordinates on + # Google Maps. The user is walked through this when running + # the ask_google script. + # + print('Loading manually cached locations...') + geo_names['googLocs'] = {} + + # geo_cache file format: + # place name|latitude coordinate|longitude coordinate + def processGeoCache(row): + geo_names['googLocs'][row[0]] = { + 'lat': row[1], + 'lng': row[2] + } + + read_tsv(Path.cwd() / 'geo_cache', processGeoCache, '|') + + return geo_names + + +def get_team_data(tba): + """Download all of the team information from The Blue Alliance.""" + print('Downloading team data from The Blue Alliance...') + return tba.teams(page=None, year=YEAR) + + +def strip_unicode(str): + """Replace unicode characters with ascii characters (e.g., replace é with e).""" + # Documentation for normalize function: + # https://docs.python.org/3/library/unicodedata.html#unicodedata.normalize + # Basically, (from what I understand) this splits the characters with accent + # marks, etc. (e.g. é) into two parts: the latin character (e.g. e) and a + # special "combining" character that represents the accent. The string is then + # encoded into ascii with the 'ignore' option, so it ignores characters that + # cannot be represented in ascii, thus removing the special combining characters + # but leaving behind the regular ones. The resulting binary is then decoded back + # into utf-8. + return (unicodedata.normalize('NFD', str) + .encode('ascii', 'ignore') + .decode('utf-8')) + + +def process_team_data(geo_names, team_data): + """ + Process all of the data that has been downloaded and write it to + teams.json and teamFullInfo.json. + """ + print('Processing and writing team info...') + + short_team_list = [] + long_short_team_list = [] + + for team in team_data: + # Skip the off-season demo teams 9970-9999 (inclusive) + if team.get('team_number') in range(9970, 10000): + continue + + # Only include the home_championship attribute for the current year + home_champ = team.get('home_championship') + if home_champ: + team['home_championship'] = home_champ.get(YEAR) + + # Get the team's city name and convert it to uppercase and ASCII (must + # be converted to ASCII because load_geonames_data loads location names + # as ASCII). Also remove leading and trailing spaces from the city name + # because sometimes the city name comes with them. + city_no_format = team.get('city') or '' + city = strip_unicode(city_no_format.upper().strip(' ')) + + # Get the country code for the team's country. + country_code = geo_names['ccodes'].get(team.get('country')) or '' + + # Get the team's state/provice/administrative division and convert to + # uppercase ASCII. + prov_no_format = team.get('state_prov') or '' + province = strip_unicode(prov_no_format.upper()) + + # Team's postal code + zip_code = team.get('postal_code') or '' + + # Needs to be uppercase (postal codes in some countries have letters) + zip_code = zip_code.upper() + + # ====== special fixes for Guam, zip weirdness, and some typos ====== + if not country_code and zip_code: + # If there is no country code, determine it by the format of the + # postal code. + if zip_code == '11073': + country_code = 'TW' + elif zip_code == '34912' or zip_code == '34469': + country_code = 'TR' + elif zip_code == '93810': + country_code = 'IL' + elif re.search('^[0-9]{4}$', zip_code): + country_code = 'AU' + elif re.search('^[0-9]{5}$', zip_code) or re.search('^[0-9]{5}-[0-9]{4}$', zip_code): + country_code = 'US' + elif re.search('^[0-9]{5}-[0-9]{3}$', zip_code): + country_code = 'BR' + elif re.search('^[A-Z][0-9][A-Z] [0-9][A-Z][0-9]$', zip_code): + country_code = 'CA' + elif re.search('^[0-9]{7}$', zip_code): + country_code = 'IL' + + if country_code == 'SE' and re.search('^[0-9]{5}', zip_code): + # For Sweden, put a space between the first three and last two + # postal code digits (e.g., 12345 becomes 123 45) + zip_code = f'{zip_code[0:3]} {zip_code[3:5]}' + + if country_code == 'US': + if province == 'GUAM': + country_code = 'GU' + elif province == 'PUERTO RICO': + country_code = 'PR' + elif city == 'NEW YORK': + city = 'NEW YORK CITY' + elif province == 'PA' and city == 'WARMINSTER': + city = 'WARMINSTER HEIGHTS' + elif province == 'MO' and city == 'LEES SUMMIT': + city = "LEE'S SUMMIT" + + if country_code == 'CL' and province == 'REGION METROPOLITANA DE SANTIAGO': + province = 'SANTIAGO METROPOLITAN' + + if country_code == 'GR' and province == 'THESSALIA': + province = 'THESSALY' + + if country_code == 'MX': + if city == 'SAN LUIS POTOTOSI': + city = 'SAN LUIS POTOSI' + if province == 'DISTRITO FEDERAL': + province = 'MEXICO CITY' + + if country_code == 'TR' and city == 'CEKMEKOY': + city = 'CEKMEKOEY' + + if country_code == 'NL' and province == 'NOORD-BRABANT': + province = 'NORTH BRABANT' + + if country_code == 'DO' and province == 'SANTO DOMINGO' and city == province: + province = 'NACIONAL' + + if country_code == 'IL': + # Israel has multiple names for administrative divisions, so this + # scraper just ignores them completely. + province = 'IL' + + if country_code == 'JP' and len(zip_code) == 7: + # For Japan, separate first three and last four digits with a dash + # (e.g., 1234567 becomes 123-4567) + f'{zip_code[0:3]}-{zip_code[3:7]}' + + if country_code == 'CA': + # special for Canada, only first three digits of zip code + zip_code = zip_code[0:3] + + if country_code == 'TW': + # This fixes certain locations in Taiwan where the province name has + # SPECIAL MUNICIPALITY or MUNICIPALITY tacked on the end some of the + # time. + if province.endswith(' SPECIAL MUNICIPALITY'): + province = province[:-len(' SPECIAL MUNICIPALITY')] + elif province.endswith(' MUNICIPALITY'): + province = province[:-len(' MUNICIPALITY')] + # ======== end of special fixes ======== + + # The latitude and longitude coordinates of the team + lat = lng = None + + # Retrieve the latitude and longitude of the team from the zip code, + # if available. + zip_country = geo_names['zipLocs'].get(country_code) + + if zip_country: + zip_loc = zip_country.get(zip_code) + + if zip_loc: + lat = zip_loc['lat'] + lng = zip_loc['lng'] + + # If the location was not retrieved... + if lat is None and country_code in geo_names['cities']: + # Retrieve the latitude and longitude of the team from the city, + # state/provice/administrative division, and country code + city_country = geo_names['cities'][country_code] + + if province in city_country: + city_prov = city_country[province] + + if city in city_prov: + city_loc = city_prov.get(city) + lat = city_loc['lat'] + lng = city_loc['lng'] + + # If the location was still not retrieved... + if lat is None: + # Get the location from cached locations that were manually + # retrieved (see googLocs section of load_geonames_data) + place = f'{city}, {province} {zip_code}, {country_code}' + goog_loc = geo_names['googLocs'].get(place) + + if goog_loc: + lat = goog_loc['lat'] + lng = goog_loc['lng'] + + # If the location was STILL not retrieved... + if lat is None: + # Notify the user that the location was not found and needs to be + # manually retrieved. Append the place to the broken_places file + # (the file is emptied at the beginning of the script). + print(f'Did not find team {team.get("key")} @ place {place}') + + with open(BROKEN_PLACES_FILE, 'a', newline="\n") as broken_places: + broken_places.write(place + '\n') + + lat = 0 + lng = 0 + + # Convert lat and lng to numbers (so they don't have quotation + # marks around them in the JSON) and round them to 3 digits after + # the decimal. + lat = round(float(lat), 3) + lng = round(float(lng), 3) + + team['lat'] = lat + team['lng'] = lng + + # Write the team number and lat/lng + short_team = { + 'team_number': team.get('team_number'), + 'lat': team.get('lat'), + 'lng': team.get('lng') + } + + short_team_list.append(short_team) + + # Write out the full team info + long_short_team = {} + + for att in TEAM_ATTRIBS: + long_short_team[att] = team.get(att) + + long_short_team_list.append(long_short_team) + + with open(Path.cwd() / 'teams.json', 'w') as out: + # When creating the JSON for the team location data, these string + # replaces put each team on its own line. It looks like this: + # [ + # # ...more teams... + # {"team_number": 404, "lat": 1.234, "lng": 5.678}, + # # ...more teams... + # ] + # This is compact but readable (making it easy to tell what changed + # when looking at a diff), but the main reason I format it this way is + # that this was the format on the previous scraper. + output = (json.dumps(short_team_list) + .replace('[{', '[\n\t{') + .replace('}, ', '},\n\t') + .replace('}]', '}\n]')) + out.write(output) + + with open(Path.cwd() / 'teamFullInfo.json', 'w') as outFull: + # For full team data, just output with standard JSON formatting + output = json.dumps(long_short_team_list, indent=4) + outFull.write(output) + + +# If the program is run with the command line argument usecache, any of the +# GeoNames files that have already been downloaded to the cache directory will +# be used instead of redownloading them. The purpose of this is to not have to +# redownload the Geonames data every time if the program has to be run several +# times in a row to resolve issues, etc. +use_cache = False + +parser = argparse.ArgumentParser() +parser.add_argument("year", + help="The year to download teams for (default: current year).", + type=int, + nargs="?") +parser.add_argument("--use-cache", + help="Do not re-download GeoNames data if already downloaded (used to save time during debugging).", + action="store_true") +results = parser.parse_args() + +tba = tbapy.TBA(AUTH_KEY) + +if results.year is None: + YEAR = datetime.now().year +else: + YEAR = results.year + +print(f"Resolving team locations for year {YEAR}...") + +get_geonames_data(results.use_cache) +geo_names = load_geonames_data() +team_data = get_team_data(tba) +process_team_data(geo_names, team_data) diff --git a/unicodes.ascii b/unicodes.ascii deleted file mode 100644 index 2012783..0000000 --- a/unicodes.ascii +++ /dev/null @@ -1,15 +0,0 @@ -\\u00c7 C -\\u00d6 O -\\u00e1 a -\\u00e3 a -\\u00e4 a -\\u00e7 c -\\u00e9 e -\\u00ed i -\\u00f3 o -\\u00f4 o -\\u00f6 o -\\u00fc u -\\u0130 I -\\u0131 i -\\u015f s