Changelog 1.2

1.2 (2015-07-15)



  • Add a database migration test from a fresh SQL structure dump.


  • 1a320a751cf: Remove observation tables.


  • #395: Move incomplete_observation logic onto colander schema.

  • #287: Replace observation models with non-db-model classes.

  • #433: Move query data validation into Query class.

  • #433: Introduce a new api.locate.query.Query class.

  • Handle any RedisError, e.g. TimeoutError and not just ConnectionErrors.

  • Update to latest raven release and update transport configuration.

  • Explicitly limit the cell cache key to its unique id parts.

  • Add fallback key to all locate responses.

  • #451: Properly test and reject empty submit requests.

  • #376: Document the added home mcc/mnc fields.

  • #419: Update geolocate docs to mention all GLS fields.



  • 2e0e620ebc92: Remove id column from content models.


  • Add workaround for andymccurdy/redis-py#633.

  • Unify v1 and v2 parse error responses to v2 format.

  • Batch key queries depending on a per-model batch size.

  • #192: Suggest a observation data retention period.

  • Optimize mapstat and station data tasks.

  • Switch to using bower for CSS/JS dependency management.

  • Update to latest versions of all CSS and JS dependencies.

  • Update to latest versions of geoip2, SQLAlchemy and unittest2.



  • 55db289fa497: Add content model composite primary keys.

  • 14dbafc4fec2: Remove new_measures indices.

  • 19d6d9fbdb6b: Increase stat table value column to biginteger.


  • Fix locate errors for incomplete cell keys.

  • Remove backwards compatibility code.



  • 38fde2949750: Remove measure_block table.


  • #287: Remove table based location_update tasks and old backup code.

  • Adjust batch sizes for new update_station tasks.

  • Bugzilla 1172833: Use apolitical names on country stats page.

  • #443: Reorganize internal module/classes.

  • Update to latest version of SQLAlchemy.



  • #446: Filter out incomplete csv cell records.

  • #287: Switch location_update tasks to new queue based system.

  • #438: Add explicit fallback choices to geolocate API.

  • Replace the last daily stats task with a queue based one.

  • #440: Allow search/locate queries without a cell id.

  • Update to latest versions of nose, simplejson and SQLAlchemy.



  • #394: Replace magic schema values by None.

  • #423: Add new public v2/geosubmit API.

  • #242: Pass through submission IP address into the data pipeline.

  • #242: Expose geoip database to async tasks.

  • Make sure there are no unexpected raven messages left after each test.

  • #434: Internal test only changes to test base classes.

  • Update to latest versions of gevent and simplejson.



  • #421: Pass through additional lookup data into the fallback query.

  • #421: Cache cell-only lookups for fallback provider queries.

  • #421: Add rate limiting to fallback provider.

  • #421: Reordered data sources to prefer fallback over geoip responses.

  • Fix api-key specific report upload counter.

  • Add workaround for raven issue #608.

  • Enable new stat counter tasks.

  • #433: Remove the wifi specific query stats.

  • Updated to latest version of alembic, celery, greenlet, kombu and pytz.



  • Correct handling for requests without API keys.

  • #421: Fix encoding of radioType in fallback queries.



  • e9c1224f6bb: Add allow_fallback column to api_key table.


  • #287: Move mapstat and score processing to separate queues/tasks.

  • #287: Keep track of uploaded data via Redis stat counters.

  • #287: Add new backup to S3 export target.

  • #421: Add fallback geolocation provider.

  • Deal with nan/inf floating point numbers in data submissions.

  • Fixed upload issues for cell entries without any radio field.

  • Updated to latest versions of certifi, greenlet, pyramid, raven and requests.



  • Allow anonymous data submission via the geosubmit API.

  • #425: Refactor internal API key logic.

  • Updated to latest raven version, requires a Sentry 7 server.

  • Updated to latests versions of billiard, pyramid and WebOb.



  • The command line invocation for the services changed, please refer to the deploy docs for the new syntax.


  • #423: Add a first version of an export job.

  • Expose all config file settings to the runtime services.

  • Move runtime related code into async/webapp sub-packages.

  • #385: Configure Python’s logging module.

  • #423: Add a new queue system using the new geosubmit v2 internal format.

  • Updated to latest versions of boto and maxminddb.



  • Make radio an internally required field.

  • Don’t validate radio fields in request side schema.

  • #418: Remove country API shortcut implementation.

  • Removed BBB code for old tasks and pre-hashkey queued values.

  • Updated to latest versions of alabaster, boto, factory_boy and pytz.



  • Remove the circus docs and example ini file.

  • Remove the vaurien/integration tests.

  • #416: Accept radioType inside the cellTowers mapping in geolocate queries.

  • Updated to latest version of Sphinx and its new dependencies.

  • Updated to latest versions of pyramid, requests, SQLAlchemy and statsd.


  • Fix unittest2 version pin.



  • 1d549c1d6cfe: Drop total_measures index on station tables.

  • 230bbf3fe044: Add index on mapstat.time column.

  • 6527bee5ac1: Remove auto-inc id columns from cell related tables.

  • 3b8d52a9eac4: Change score, stat and measure_block enum columns to tinyint.


  • Replace heka-py-raven with a direct raven client.

  • #319: Remove the per station ingress filtering.

  • Allow partial cell ids in geolocate/geosubmit APIs.

  • Removed the mixed locate/submit mode from the geosubmit API.

  • #402: Avoid multiple validation of common report data fields.

  • Add a new CellCountryProvider to allow country searches based on cell data.

  • #406: Allow access to the country API via empty GET requests.

  • Massive internal code refactoring and cleanup.

  • Updated to latest versions of iso3166, pyramid and requests.



  • Reestablish database connections on connection failures.



  • Backup/delete all observation data except for the current day.

  • Updated to latest versions of boto, Chameleon, gunicorn, jaraco.util, Mako, psutil, Pygments, pyzmq and WebTest.



  • Specify statsd prefix in application code instead of heka config.

  • Fix geoip country lookup for entries without countries.

  • #274: Extend monitor view to include geoip db status.



  • 10542c592089: Remove invalid lac values.

  • fe2cfea89f5: Change cell/_blacklist tables primary keys.


  • #367: Tighten lac filtering to exclude 65534 (gsm) and 65535 (all).

  • Remove alembic migrations before the 1.0 PyPi release.

  • #353: Remove auto-inc id column from cell/_blacklist tables.

  • Add additional stats to judge quality of WiFi based queries.

  • #390: Remove command line importer script.



  • 188e749e51ec: Change lac/cid columns to signed integers.


  • #352: Switch to new maxmind v2 database format and libraries.

  • #274: Add a new __monitor__ endpoint.

  • #291: Allow 32bit UMTS cell ids, tighten checks for CDMA and LTE.

  • #311: On station creation optionally use previous blacklist time.

  • #378: Use colander for internal data validation.

  • Remove explicit queue declaration from celery base task.

  • Updated to latest versions of alembic, boto, Chameleon, jaraco.util, mobile-codes, psutil, requests-mock, WSGIProxy2 and zope.deprecation.



  • 48ab8d41fb83: Move cell areas into separate table.


  • Prevent non-countries from being returned by the country API.

  • #368: Add per API key metrics for uploaded batches, reports and observations.

  • Clarify metric names related to batches/reports/observations, add new items.uploaded.batch_size pseudo-timer and items.uploaded.reports counter.

  • Introduce new internal GeoIPWrapper.country_lookup API.

  • #343: Fall back to GeoIP for incomplete search requests.

  • #349/#350: Move cell areas into new table.

  • Give all celery queues a prefix to better distinguish them in Redis.

  • #354: Remove scan_lacs fallback code looking at new_measures.

  • Updated to latest versions of alembic, argparse, billiard, celery, colander, filechunkio, iso8601, kombu, PyMySQL, pytz, requests, six, WebTest and zope.interface.


  • #371: Add new country API.


  • Add api key specific stats to count best data lookup hits/misses.

  • Validate WiFi data in location lookups earlier in the process.

  • #312: Add email field to User model.

  • #287: Move lac update scheduling to Redis based queue.

  • #304: Auto-correct radio field of GSM cells with large cid values.

  • Move responsibility for lac entry deletion into update_lac task.

  • Accept more ASU values but tighten signal strength validation.

  • #305: Stricter range check for mnc values for non-CDMA networks.

  • Add a convenience session.on_post_commit helper method.

  • #17: Remove the unused code for cell backfill.

  • #41: Explicitly allow anonymous data submissions.

  • #335: Omit incomplete cell records from exports.

  • Delete measures in batches of 10k rows in backup tasks.

  • Re-arrange backup tasks to avoid holding db session open for too long.

  • Report errors for malformed data in submit call to sentry.

  • Report errors during backup job to sentry.

  • #332: Fix session handling in map tiles generation.

  • Updated to latest versions of argparse, Chameleon, irc, Pygments, pyramid, translationstring and unittest2.


  • #330: Expand api keys and download sections.

  • Close database session early in map tiles generation.

  • Close database session early in export task to avoid timeout errors while uploading data to S3.

  • Optimize cell export task and avoid datetime/unixtime conversions.

  • Add an index on cell.modified to speed up cell export task.

  • Updated to latest versions of boto, irc, pygeoip, pytz, pyzmq, simplejson and unittest2.


  • Add play store link for Mozilla Stumbler to apps page.

  • Updated privacy notice style to match general Mozilla style.

  • Switch gunicorn to use a gevent-based worker.

  • Clean last database result from connections on pool checkin.

  • Close the database connections even if exceptions occurred.