The companion to the Phone Your Rep frontend.
Data sources:
Congress - https://github.com/TheWalkers/congress-legislators and https://github.com/unitedstates
State and district shapefiles - https://www.census.gov/geo/maps-data/data/tiger-cart-boundary.html
ZCTA to congressional district relationship file - http://www2.census.gov/geo/docs/maps-data/data/rel/zcta_cd111_rel_10.txt
State Reps - http://openstates.org
If you are too busy to do the manual installation, you can download a Vagrant BOX which has the requirements below already installed, download it here.
https://s3.amazonaws.com/debugpyr/pyr.box
rbenv install 2.4.1 or rvm install 2.4.1
Make sure you have PostgreSQL installed, and then install the PostGIS extension. If you're using MacOS you can try installing with Homebrew. Otherwise it's recommended that you use the Vagrant box.
$ brew install postgres
$ brew install postgis
Mac users can also download the Heroku PostgreSQL app instead of running brew install. The app comes with the PostGIS extension enabled and everything working out of the box. This is by far the easiest route to get PostGIS on a Mac.
Then
$ gem install bundler
$ bundle install
$ bundle exec rake db:pyr:setup
Run the tests
$ bundle exec rake
If you didn't get any errors during setup and the tests are passing, you can seed the database. It will take a little while, so grab a cold one
$ bundle exec rake db:seed
If you've already configured and seeded the database before and just need to update to the most current data, skip ahead to #Updating. If you want to reset the database and seed with one command run bundle exec rake db:pyr:setup_and_seed
If you're configuring for the first time and you're getting errors, or you don't want to do a complete reset, or you're some kind of control freak, here are the manual steps broken down
$ bundle exec rake db:drop # skip this unless you're resetting
$ bundle exec rake db:create
$ bundle exec rake db:gis:setup # enables the PostGIS extension
$ bundle exec rake db:migrate
Migrating is your first test that you have a properly configured database. If you get errors while migrating, you may have PostGIS configuration issues and your database is not recognizing the geospatial datatypes. Read up on the documentation for RGeo and ActiveRecord PostGIS Adapter to troubleshoot.
You will need a properly provisioned Postgres database with PostGIS enabled. On Heroku you'll need a minimum of hobby-basic tier database and follow these steps.
Your Heroku buildpacks should be in the following order prior to your first git push:
https://github.com/diowa/heroku-buildpack-rgeo-prep.git
https://github.com/peterkeen/heroku-buildpack-vendorbinaries.git
heroku/ruby
Git push and run this command to load the schema into the database:
$ heroku run --app [APP_NAME] DISABLE_DATABASE_ENVIRONMENT_CHECK=1 rake db:schema:load
You'll also want to configure the following environment variables:
# necessary for performing geocoding. You can configure another geocoding service and API key in config/initializers/geocoder.rb
GOOGLE_API_KEY = your_key
LD_LIBRARY_PATH = /app/lib
# Necessary for running update tasks on state rep data
OPENSTATES_API_KEY = your_key
# AWS bucket for QR code images. Set this var if you want to customize the images and manage your own bucket. Defaults to phone-your-rep-images if you don't
PYR_S3_BUCKET = your_bucket
# Governator gem uses these to scrape data on U.S. governors. This feature is deprecated in favor of scraping data from the open source [CivilServiceUSA/us-governors repo](https://github.com/CivilServiceUSA/us-governors)
TWITTER_ACCESS_TOKEN = your_token
TWITTER_ACCESS_TOKEN_SECRET = your_secret_token
TWITTER_CONSUMER_KEY = your_key
TWITTER_CONSUMER_SECRET = your_secret_key
Many of the offices have coordinates preloaded in the seed data. Any that don't will automatically be geocoded during seeding.
The geocoder gem allows you to do some geocoding without an API key. It will probably be enough for seeding and development. However, if you want to use your own API key for geocoding, you can configure it in config/initializers/geocoder.rb. You will also need to check this file for deployment, as it's configured to access an environment variable for the API key in production.
If you don't want to geocode any of the offices at all, comment out this line in office_location.rb
after_validation :geocode, if: :needs_geocoding?The seeds.rb file invokes a handful of discreet seeding tasks. If you want to isolate any of these, or seed manually, here they are broken down:
$ bundle exec rake db:pyr:seed_states
$ bundle exec rake db:pyr:seed_districts
The two tasks above load the basic state and district data such as names and codes.
$ bundle exec rake db:pyr:shapefiles
The shapefiles task loads the geographic boundary data for the states and districts, and is the last test that your database is configured properly for GIS.
$ bundle exec rake db:pyr:seed_reps
The seed_reps task loads all of the rep and office location data.
If you want to be able to look up congressional districts by ZCTA, run bundle exec rake db:pyr:zctas, or if starting from scratch you can pass zctas=true as a variable to the seeds task, e.g.
$ bundle exec rake db:seed zctas=true
or
$ bundle exec rake db:pyr:setup_and_seed zctas=true
If you want to scrape all state reps and load them into the database run the following command:
$ bundle exec rake db:pyr:update:state_reps
This will take some time to complete. Maybe an hour?
Finally
$ bundle exec rails s
The app is configured to get QR code images from the phone-your-rep-images S3 bucket by default. These QR codes are kept up to date with the current data. If you wish to generate your own, you can do so easily by following these steps:
Then just
# If you want to generate and upload QR codes for all congressional and state reps, plus governors
$ bundle exec rake pyr:qr_codes:create_all
# If you want to focus on smaller sets:
$ bundle exec rake pyr:qr_codes:create[congress]
$ bundle exec rake pyr:qr_codes:create[governors]
$ bundle exec rake pyr:qr_codes:create[vt] # Substitute argument with any two-letter state and U.S. territory abbreviation
This will generate the images, empty the bucket, upload the images, and then delete the local copies. If you set the environment variable properly, your app should automatically point to the right URLs.
Generating the QR codes for all reps takes a couple hours to execute and is CPU intensive.
If you just need to update your existing database with the most current data then run
$ bundle exec rake db:pyr:update:all
The discreet steps are broken down as follows:
$ bundle exec rake db:pyr:update:retired_reps
This deactivates any reps (and their office locations) that are no longer serving in congress.
$ bundle exec rake db:pyr:update:current_reps
This updates basic info for the active reps, including name, role, state, district, and DC office. New reps will be added to the database.
$ bundle exec rake db:pyr:update:socials
This updates the social media accounts for active reps.
$ bundle exec rake db:pyr:update:office_locations
This updates all of the active district offices for all reps, adds new ones, and deactivates those no longer in service. Updated VCards are also generated for each office.
$ bundle exec rake db:pyr:update:photos
Verifies and updates photo urls for all reps.
If you need to generate updated QR codes you can run the update command as rake db:pyr:update:all qr_codes=true
All of the raw data is stored in the code base as YAML files which track files from unitedstates and TheWalkers. Each database update task automatically updates the YAML file first by curl-ing its online source and committing any changes. These files may be updated often, so it's recommended that you check for updates once a week or so.
The full database update is a little time consuming. Fetching the raw data first can let you check for changes and save you from running the entire database update task if there are no changes made.
You can update all of the local YAML data files and commit the changes without updating the database by running
rake db:pyr:update:raw_data
This can be broken down into separate tasks for the individual files:
rake db:pyr:update:fetch_retired_reps
rake db:pyr:update:fetch_current_reps
rake db:pyr:update:fetch_socials
rake db:pyr:update:fetch_office_locations
For any of the individual update tasks (exlcuding db:pyr:update:all) you can specify a file destination other than the default by setting it in the file variable e.g. rake db:pyr:update:socials file=socials.yaml. This will download the data to socials.yaml in the root directory, commit the change, and update the database from that file, leaving the default file (lib/seeds/legislators-social-media.yaml) unchanged.
You can also specify an alternative data source (as long as it's in YAML format) by setting it as the source variable e.g. rake db:pyr:update:socials source=https://www.yoursource.com/data.yaml
This is deployed on Heroku. See the Installation section for deployment steps.
Contributions are encouraged! Write test coverage for all new code and commit your changes to a feature branch. Make sure the build passes with bundle exec rake, and fix any broken tests and Rubocop style issues. Then submit a pull request!
Ruby developers can try the pyr gem
This API is in beta. An example request to the API looks like this:
https://phone-your-rep.herokuapp.com/api/beta/reps?lat=42.3134848&long=-71.2072321
And here is the response:
{
"total_records": 3,
"_links": {
"self": {
"href": "https://phone-your-rep.herokuapp.com/api/beta/reps?lat=42.3134848&long=-71.2072321"
}
},
"reps": [
{
"self": "https://phone-your-rep.herokuapp.com/api/beta/reps/M000133",
"state": {
"self": "https://phone-your-rep.herokuapp.com/states/25",
"state_code": "25",
"name": "Massachusetts",
"abbr": "MA"
},
"bioguide_id": "M000133",
"official_full": "Edward J. Markey",
"role": "United States Senator",
"party": "Democrat",
"senate_class": "02",
"last": "Markey",
"first": "Edward",
"middle": "J.",
"nickname": "Ed",
"suffix": null,
"contact_form": "http://www.markey.senate.gov/contact",
"url": "http://www.markey.senate.gov",
"photo": "https://theunitedstates.io/images/congress/450x550/M000133.jpg",
"twitter": "SenMarkey",
"facebook": "EdJMarkey",
"youtube": "RepMarkey",
"instagram": null,
"googleplus": null,
"twitter_id": "3047090620",
"facebook_id": "6846731378",
"youtube_id": "UCT1ujew5yQy2uMhGrjiKHoA",
"instagram_id": null,
"office_locations": [
{
"self": "https://phone-your-rep.herokuapp.com/api/beta/office_locations/1314",
"id": 1314,
"bioguide_id": "M000133",
"office_type": "district",
"distance": 8.2,
"building": "975 JFK Federal Building",
"address": "15 Sudbury St.",
"suite": "",
"city": "Boston",
"state": "MA",
"zip": "02203",
"phone": "617-565-8519",
"fax": "",
"hours": "",
"latitude": 42.3613091,
"longitude": -71.0593927,
"v_card_link": "https://phone-your-rep.herokuapp.com/v_cards/1314",
"qr_code_link": "https://s3.amazonaws.com/phone-your-rep-images/9o6tqptj1_Markey_district_1314.png"
},
{
"self": "https://phone-your-rep.herokuapp.com/api/beta/office_locations/1315",
"id": 1315,
"bioguide_id": "M000133",
"office_type": "district",
"distance": 42.5,
"building": "",
"address": "222 Milliken Blvd.",
"suite": "Suite 312",
"city": "Fall River",
"state": "MA",
"zip": "02721",
"phone": "508-677-0523",
"fax": "",
"hours": "",
"latitude": 41.6999176,
"longitude": -71.1587266,
"v_card_link": "https://phone-your-rep.herokuapp.com/v_cards/1315",
"qr_code_link": "https://s3.amazonaws.com/phone-your-rep-images/3c4hp2pyzd_Markey_district_1315.png"
},
{
"self": "https://phone-your-rep.herokuapp.com/api/beta/office_locations/1316",
"id": 1316,
"bioguide_id": "M000133",
"office_type": "district",
"distance": 72.4,
"building": "",
"address": "1550 Main St.",
"suite": "4th Floor",
"city": "Springfield",
"state": "MA",
"zip": "01101",
"phone": "413-785-4610",
"fax": "",
"hours": "",
"latitude": 42.1032165,
"longitude": -72.5929441,
"v_card_link": "https://phone-your-rep.herokuapp.com/v_cards/1316",
"qr_code_link": "https://s3.amazonaws.com/phone-your-rep-images/77buuaccxv_Markey_district_1316.png"
},
{
"self": "https://phone-your-rep.herokuapp.com/api/beta/office_locations/200",
"id": 200,
"bioguide_id": "M000133",
"office_type": "capitol",
"distance": 385.0,
"building": null,
"address": "255 Dirksen Senate Office Building",
"suite": null,
"city": "Washington",
"state": "DC",
"zip": "20510",
"phone": "202-224-2742",
"fax": null,
"hours": null,
"latitude": 38.8928318,
"longitude": -77.0043625,
"v_card_link": "https://phone-your-rep.herokuapp.com/v_cards/200",
"qr_code_link": "https://s3.amazonaws.com/phone-your-rep-images/5puc375jgq_Markey_capitol_200.png"
}
]
},
{
"self": "https://phone-your-rep.herokuapp.com/api/beta/reps/W000817",
"state": {
"self": "https://phone-your-rep.herokuapp.com/states/25",
"state_code": "25",
"name": "Massachusetts",
"abbr": "MA"
},
"bioguide_id": "W000817",
"official_full": "Elizabeth Warren",
"role": "United States Senator",
"party": "Democrat",
"senate_class": "01",
"last": "Warren",
"first": "Elizabeth",
"middle": null,
"nickname": null,
"suffix": null,
"contact_form": "http://www.warren.senate.gov/?p=email_senator#thisForm",
"url": "http://www.warren.senate.gov",
"photo": "https://theunitedstates.io/images/congress/450x550/W000817.jpg",
"twitter": "SenWarren",
"facebook": "senatorelizabethwarren",
"youtube": "senelizabethwarren",
"instagram": null,
"googleplus": null,
"twitter_id": "970207298",
"facebook_id": "131559043673264",
"youtube_id": "UCTH9zV8Imw09J5bOoTR18_A",
"instagram_id": null,
"office_locations": [
{
"self": "https://phone-your-rep.herokuapp.com/api/beta/office_locations/1897",
"id": 1897,
"bioguide_id": "W000817",
"office_type": "district",
"distance": 8.2,
"building": "2400 JFK Federal Building",
"address": "15 Sudbury St.",
"suite": "",
"city": "Boston",
"state": "MA",
"zip": "02203",
"phone": "617-565-3170",
"fax": "",
"hours": "",
"latitude": 42.3613091,
"longitude": -71.0593927,
"v_card_link": "https://phone-your-rep.herokuapp.com/v_cards/1897",
"qr_code_link": "https://s3.amazonaws.com/phone-your-rep-images/9g9zrqnpbu_Warren_district_1897.png"
},
{
"self": "https://phone-your-rep.herokuapp.com/api/beta/office_locations/1898",
"id": 1898,
"bioguide_id": "W000817",
"office_type": "district",
"distance": 72.4,
"building": "",
"address": "1550 Main St.",
"suite": "Suite 406",
"city": "Springfield",
"state": "MA",
"zip": "01103",
"phone": "413-788-2690",
"fax": "",
"hours": "",
"latitude": 42.1032165,
"longitude": -72.5929441,
"v_card_link": "https://phone-your-rep.herokuapp.com/v_cards/1898",
"qr_code_link": "https://s3.amazonaws.com/phone-your-rep-images/61252tqznk_Warren_district_1898.png"
},
{
"self": "https://phone-your-rep.herokuapp.com/api/beta/office_locations/367",
"id": 367,
"bioguide_id": "W000817",
"office_type": "capitol",
"distance": 385.0,
"building": null,
"address": "317 Hart Senate Office Building",
"suite": null,
"city": "Washington",
"state": "DC",
"zip": "20510",
"phone": "202-224-4543",
"fax": null,
"hours": null,
"latitude": 38.8928318,
"longitude": -77.0043625,
"v_card_link": "https://phone-your-rep.herokuapp.com/v_cards/367",
"qr_code_link": "https://s3.amazonaws.com/phone-your-rep-images/4cb6hk2egn_Warren_capitol_367.png"
}
]
},
{
"self": "https://phone-your-rep.herokuapp.com/api/beta/reps/K000379",
"state": {
"self": "https://phone-your-rep.herokuapp.com/states/25",
"state_code": "25",
"name": "Massachusetts",
"abbr": "MA"
},
"district": {
"self": "https://phone-your-rep.herokuapp.com/districts/2504",
"full_code": "2504",
"code": "04",
"state_code": "25"
},
"bioguide_id": "K000379",
"official_full": "Joseph P. Kennedy III",
"role": "United States Representative",
"party": "Democrat",
"senate_class": null,
"last": "Kennedy",
"first": "Joseph",
"middle": "P.",
"nickname": null,
"suffix": "III",
"contact_form": null,
"url": "https://kennedy.house.gov",
"photo": "https://theunitedstates.io/images/congress/450x550/K000379.jpg",
"twitter": "RepJoeKennedy",
"facebook": "301936109927957",
"youtube": null,
"instagram": "repkennedy",
"googleplus": null,
"twitter_id": "1055907624",
"facebook_id": "301936109927957",
"youtube_id": "UCgfHlaGqxD8p-2V_YlNIqrA",
"instagram_id": "1328567154",
"office_locations": [
{
"self": "https://phone-your-rep.herokuapp.com/api/beta/office_locations/1200",
"id": 1200,
"bioguide_id": "K000379",
"office_type": "district",
"distance": 2.9,
"building": "",
"address": "29 Crafts St.",
"suite": "Suite 375",
"city": "Newton",
"state": "MA",
"zip": "02458",
"phone": "617-332-3333",
"fax": "617-332-3308",
"hours": "M-F 9-5:30PM",
"latitude": 42.3548224,
"longitude": -71.1999166,
"v_card_link": "https://phone-your-rep.herokuapp.com/v_cards/1200",
"qr_code_link": "https://s3.amazonaws.com/phone-your-rep-images/7gubdgo2kr_Kennedy_district_1200.png"
},
{
"self": "https://phone-your-rep.herokuapp.com/api/beta/office_locations/1199",
"id": 1199,
"bioguide_id": "K000379",
"office_type": "district",
"distance": 25.8,
"building": "",
"address": "8 N. Main St.",
"suite": "Suite 200",
"city": "Attleboro",
"state": "MA",
"zip": "02703",
"phone": "508-431-1110",
"fax": "508-431-1101",
"hours": "M-F 9-5:30PM",
"latitude": 41.9449626,
"longitude": -71.2846799,
"v_card_link": "https://phone-your-rep.herokuapp.com/v_cards/1199",
"qr_code_link": "https://s3.amazonaws.com/phone-your-rep-images/3yratflx0b_Kennedy_district_1199.png"
},
{
"self": "https://phone-your-rep.herokuapp.com/api/beta/office_locations/368",
"id": 368,
"bioguide_id": "K000379",
"office_type": "capitol",
"distance": 385.4,
"building": null,
"address": "434 Cannon HOB",
"suite": null,
"city": "Washington",
"state": "DC",
"zip": "20515-2104",
"phone": "202-225-5931",
"fax": null,
"hours": null,
"latitude": 38.8870943,
"longitude": -77.0082254,
"v_card_link": "https://phone-your-rep.herokuapp.com/v_cards/368",
"qr_code_link": "https://s3.amazonaws.com/phone-your-rep-images/62qf7ihqu3_Kennedy_capitol_368.png"
}
]
}
]
}