Skip to main content
SpringerLink
Log in

Assessing the validity of facilitated-volunteered geographic information: comparisons of expert and novice ratings

  • Published:
GeoJournal Aims and scope Submit manuscript

Abstract

Facilitated-voluntary geographic information (f-VGI) is a promising method to enable systematic collection of data from residents about their physical and social environment. The method capitalizes on ubiquitous mobile smartphones to empower collection of geospatially-referenced data. It is important to evaluate the validity of user-generated content for use in research or program planning. The purpose of this study was to test whether the aggregated environmental (“bikeability”) ratings from novice community residents converges with ratings from experts using a robust research-based, paper audit-tool (the established Pedestrian Environment Data Scan (PEDS) tool). Equivalence testing statistically showed overall agreement between the composite ratings of bikeability within the novice group. Agreement in categorical ratings between novices and experts were examined using the summary agreement index, which showed substantial agreement across the 10 locations rated by 11 novices using an f-VGI mobile application and four experts using PEDS; variability depended on the nature of the specific questions asked. Results reveal overall substantial agreement between novice and expert ratings for both composite scores and individual categorical ratings. However, additional research is needed to refine the methodology for use in formalized research applications.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  • Berkowitz, B. (2010). Using GPS to tag potholes. http://en.seeclickfix.com/.

  • Berrigan, D., Troiano, R. P., McNeel, T., DiSogra, C., & Ballard-Barbash, R. (2006). Active transportation increases adherence to activity recommendations. American Journal of Preventive Medicine, 31(3), 210–216.

    Article  Google Scholar 

  • Boarnet, M. G., Day, K., Alfonzo, M., Forsyth, A., & Oakes, M. (2006). The Irvine–Minnesota inventory to measure built environments: Reliability tests. American Journal of Preventive Medicine, 30(2), 153–159. doi:10.1016/j.amepre.2005.09.018.

    Article  Google Scholar 

  • Bordass, B., & Leaman, A. (2005). Phase 5: Occupancy - post-occupancy evaluation. In W. F. E. Preiser & J. Vischer (Eds.), Assessing building performance (pp. 72–78). Oxford: Elsevier.

    Google Scholar 

  • Brabham, D. C. (2008). Crowdsourcing as a model for problem solving an introduction and cases. Convergence: The International Journal of Research Into New Media Technologies, 14(1), 75–90.

    Article  Google Scholar 

  • Brownson, R., Hoehner, C., Day, K., Forsyth, A., & Sallis, J. (2009). Measuring the built environment for physical activity: State of the science. American Journal of Preventive Medicine, 36(4 Suppl), S123–S599.

    Google Scholar 

  • Buman, M. P., Winter, S. J., Sheats, J. L., Hekler, E. B., Otten, J. J., Grieco, L. A., et al. (2013). The Stanford Healthy Neighborhood Discovery Tool: A computerized tool to assess active living environments. Amercian Journal of Preventative Medicine, 44(4), e41–e47. doi:10.1016/j.amepre.2012.11.028.

    Article  Google Scholar 

  • Chatzimilioudis, G., Konstantinidis, A., Laoudias, C., & Zeinalipour-Yazti, D. (2012). Crowdsourcing with smartphones. IEEE Internet Computing, 16(5), 36–44. doi:10.1109/MIC.2012.70.

    Article  Google Scholar 

  • Clifton, K. J., Livi Smith, A. D., & Rodriguez, D. (2007). The development and testing of an audit for the pedestrian environment. Landscape and Urban Planning, 80(1–2), 95–110. doi:10.1016/j.landurbplan.2006.06.008.

    Article  Google Scholar 

  • Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46.

    Article  Google Scholar 

  • Comber, A., See, L., Fritz, S., Van der Velde, M., Perger, C., & Foody, G. (2013). Using control data to determine the reliability of volunteered geographic information about land cover. International Journal of Applied Earth Observation and Geoinformation, 23(1), 37–48. doi:10.1016/j.jag.2012.11.002.

    Article  Google Scholar 

  • Cunningham, G. O., Michael, Y. L., Farquhar, S. A., & Lapidus, J. (2005). Developing a reliable senior walking environmental assessment tool. American Journal of Preventive Medicine, 29(3), 215–217. doi:10.1016/j.amepre.2005.05.002.

    Article  Google Scholar 

  • Dickinson, J., & Bonney, R. (Eds.). (2012). Citizen science: Public participation in environmental research. Ithaca: Cornell University Press.

    Google Scholar 

  • Elwood, S., Goodchild, M. F., & Sui, D. Z. (2012). Researching volunteered geographic information: Spatial data, geographic research, and new social practice. Annals of the Association of American Geographers, 102(3), 571–590. doi:10.1080/00045608.2011.595657.

    Article  Google Scholar 

  • Emery, J., Crump, C., & Bors, P. (2003). Reliability and validity of two instruments designed to assess the walking and bicycling suitability of sidewalks and roads. American Journal of Health Promotion, 18(1), 38–46. doi:10.4278/0890-1171-18.1.38.

    Article  Google Scholar 

  • Ewing, R., Brownson, R. C., & Berrigan, D. (2006). Relationship between urban sprawl and weight of United States youth. American Journal of Preventive Medicine, 31(6), 464–474.

    Article  Google Scholar 

  • Feng, J., Glass, T., Curriero, F., Stewart, W., & Schwartz, B. (2010). The built environment and obesity: A systematic review of the epidemiologic evidence. Health Place, 16(2), 175–190.

    Article  Google Scholar 

  • Fienen, M. N., & Lowry, C. S. (2012). Social. Water—A crowdsourcing tool for environmental data acquisition. Computers & Geosciences, 49, 164–169.

    Article  Google Scholar 

  • Flanagin, A., & Metzger, M. (2008). The credibility of volunteered geographic information. GeoJournal, 72(3–4), 137–148. doi:10.1007/s10708-008-9188-y.

    Article  Google Scholar 

  • Flegal, K., Carroll, M., Ogden, C., & Curtin, L. (2010). Prevalence and trends in obesity among US adults, 1999–2008. Journal of the American Medical Association, 303(3), 235–241.

    Article  Google Scholar 

  • Fleiss, J. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378–382.

    Article  Google Scholar 

  • Fleiss, J., & Cohen, J. (1973). The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and Psychological Measurement, 33(3) 613–619.

    Article  Google Scholar 

  • Foody, G. M., & Boyd, D. S. (2012). Exploring the potential role of volunteer citizen sensors in land cover map accuracy assessment. In 10th international symposium on spatial accuracy assessment in natural resources and environmental science, Florianopolis, Brazil, Jul 10–13 2012 (pp. 203–208).

  • Frank, L., Kerr, J., Sallis, J., Miles, R., & Chapman, J. (2008). A hierarchy of sociodemographic and environmental correlates of walking and obesity. Amercian Journal of Preventative Medicine, 47(2), 172–178.

    Article  Google Scholar 

  • Fritz, S., See, L., McCallum, I., Schill, C., Perger, C., & Obersteiner, M. (2011). Building a crowd-sourcing tool for the validation of urban extent and gridded population. In B. Murgante, O. Gervasi, A. Iglesias, D. Taniar, B. O.Apduhan (Eds.), Computational science and its applications-ICCSA 2011 (pp. 39–50). Springer.

  • Goodchild, M. (2007). Citizens as sensors: The world of volunteered geography. GeoJournal, 69(4), 211–221. doi:10.1007/s10708-007-9111-y.

    Article  Google Scholar 

  • Goodchild, M., & Li, L. (2012). Assuring the quality of volunteered geographic information. Spatial Statistics, 1(1), 110–120. doi:10.1016/j.spasta.2012.02.002.

    Article  Google Scholar 

  • Gordon-Larsen, P., Nelson, M., Page, P., & Popkin, B. (2006). Inequality in the built environment underlies key health disparities in physical activity and obesity. Pediatrics, 117(2), 417–424.

    Article  Google Scholar 

  • Heinrich, K., Lee, R., Regan, G., Reese-Smith, J., Howard, H., Haddock, C., et al. (2008). How does the built environment relate to body mass index and obesity prevalence among public housing residents? Amercian Journal of Health Promotion, 22(3), 187–194.

    Article  Google Scholar 

  • Heipke, C. (2010). Crowdsourcing geospatial data. ISPRS Journal of Photogrammetry and Remote Sensing, 65(6), 550–557. doi:10.1016/j.isprsjprs.2010.06.005.

    Article  Google Scholar 

  • Humpel, N., Owen, N., & Leslie, E. (2002). Environmental factors associated with adults’ participation in physical activity: A review. American Journal of Preventive Medicine, 22(3), 188–199. doi:10.1016/S0749-3797(01)00426-3.

    Article  Google Scholar 

  • Keast, E., Carlson, N., Chapman, N., & Michael, Y. (2010). Using built environmental observation tools: Comparing two methods of creating a measure of the built environment. American Journal of Health Promotion, 24(5), 354–361. doi:10.4278/ajhp.08063-QUAN-81.

    Article  Google Scholar 

  • Koriakine, A., & Saveliev, E. (2016). Wikimapia. http://www.wikimapia.org/.

  • Kraemer, H. C. (1981). Intergroup concordance: Definition and estimation. Biometrika, 68(3), 641–646. doi:10.1093/biomet/68.3.641.

    Article  Google Scholar 

  • Krippendorff, K. (1970). Estimating the reliability, systematic error, and random error of interval data. Educational and Psychological Measurement, 30(1), 61–70.

    Article  Google Scholar 

  • Kwarteng, J., Schulz, A. J., Mentz, G., Zenk, S. N., & Opperman, A. (2014). Associations between observed neighborhood characteristics and physical activity: Findings from a multiethnic urban community. Journal of Public Health, 36(3), 358–367.

    Article  Google Scholar 

  • Lee, R. E., McAlexander, K., & Banda, J. (2011). Reversing the obesogenic environment. Champaign: Human Kinetics.

    Google Scholar 

  • Lytle, L. A. (2009). Measuring the food environment: State of the science. American Journal of Preventive Medicine, 36(4), S134–S144.

    Article  Google Scholar 

  • McKinnon, R. A., Reedy, J., Morrissette, M. A., Lytle, L. A., & Yaroch, A. L. (2009). Measures of the food environment: A compilation of the literature, 1990–2007. American Journal of Preventive Medicine, 36(4), S124–S133.

    Article  Google Scholar 

  • Moudon, A. V., & Lee, C. (2003). Walking and bicycling: An evaluation of environmental audit instruments. American Journal of Health Promotion, 18(1), 21–37. doi:10.4278/0890-1171-18.1.21.

    Article  Google Scholar 

  • Ogden, C. L., Lamb, M. M., Carroll, M. D., & Flegal, K. M. (2010). Obesity and socioeconomic status in adults: United States, 2005–2008. NCHS Data Brief, 50(51), 1–8.

    Google Scholar 

  • Pikora, T. J., Bull, F. C. L., Jamrozik, K., Knuiman, M., Giles-Corti, B., & Donovan, R. J. (2002). Developing a reliable audit instrument to measure the physical environment for physical activity. American Journal of Preventive Medicine, 23(3), 187–194. doi:10.1016/S0749-3797(02)00498-1.

    Article  Google Scholar 

  • Robinson, A. P., Duursma, R. A., & Marshall, J. D. (2005). A regression-based equivalence test for model validation: Shifting the burden of proof. Tree physiology, 25(7), 903–913.

  • Saelens, B. E., Sallis, J. F., Black, J. B., & Chen, D. (2003). Neighborhood-based differences in physical activity: An environment scale evaluation. American Journal of Public Health, 93(9), 1552–1558.

    Article  Google Scholar 

  • Sallis, J., Bauman, A., & Pratt, M. (1998). Environmental and policy interventions to promote physical activitya. American Journal of Preventive Medicine, 15(4), 379–397. doi:10.1016/S0749-3797(98)00076-2.

    Article  Google Scholar 

  • Schlossberg, M. (2006). From TIGER to audit instruments: Measuring neighborhood walkability with street data based on geographic information systems. Transportation Research Record: Journal of the Transportation Research Board 1982(1), 48–56.

  • Schlossberg, M., Agrawal, A. W., & Irvin, K. (2007). An assessment of GIS-enabled walkability audits. URISA Journal, 19(2), 5–11.

    Google Scholar 

  • Schlossberg, M., Evers, C., Kato, K., & Brehm, C. (2012). Active transportation, citizen engagement and livability: Coupling citizens and smartphones to make the change. URISA Journal, 25(2), 61–70.

    Google Scholar 

  • Schlossberg, M., Johnson-Shelton, D., Evers, C., & Moreno-Black, G. (2015). Refining the grain: Using resident-based walkability audits to better understand walkable urban form. Journal of Urbanism: International Research on Placemaking and Urban Sustainability, 8(3), 260–278.

    Google Scholar 

  • Seeger, C. (2008). The role of facilitated volunteered geographic information in the landscape planning and site design process. GeoJournal, 72(3–4), 199–213. doi:10.1007/s10708-008-9184-2.

    Article  Google Scholar 

  • Seeger, C. (2017). Volunteered geographic information system in planning. In C. Yamu, A. Poplin, O. Davisch, & G. De Roo (Eds.), The virtual and the real in planning and urban design: Perspectives, practices and applications. Abingdon: Routledge.

    Google Scholar 

  • Seeger, C., Lillehoj, C., Wilson, S., & Jensen, A. (2014). Facilitated-VGI, smartphones and geodesign: Building a coalition while mapping community infrastructure. In U. WissenHayek, P. Fricker, & E. Buhmann (Eds.), Digital landscape architecture (pp. 300–308). Berlin: Herbert Wichmann Verlag.

    Google Scholar 

  • Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420.

    Article  Google Scholar 

  • Stevens, M., & D’Hondt, E. (2010). Crowdsourcing of pollution data using smartphones. In Workshop on ubiquitous crowdsourcing.

  • Story, M., Giles-Corti, B., Yaroch, A. L., Cummins, S., Frank, L. D., Huang, T. T.-K., et al. (2009). Work group IV: Future directions for measures of the food and physical activity environments. American Journal of Preventive Medicine, 36(4), S182–S188.

    Article  Google Scholar 

  • Sui, D., Elwood, S., & Goodchild, M. (2012). Crowdsourcing geographic knowledge: Volunteered geographic information (VGI) in theory and practice. Berlin: Springer Science & Business Media.

    Google Scholar 

  • Surowiecki, J. (2005). The wisdom of crowds. New York, NY: Anchor Books.

    Google Scholar 

  • Sweeney, G., Hand, M., Kaiser, M., Clark, J. K., Rogers, C., & Spees, C. (2015). The state of food mapping academic literature since 2008 and review of online GIS-based food mapping resources. Journal of Planning Literature, 31(2), 123–219. doi:10.1177/0885412215599425

  • Turner, S., Hottenstein, A., & Shunk, G. (1997). Bicycle and pedestrian travel demand forecasting: Literature review. College Station: Texas Transportation Institute, Texas A & M University System.

    Google Scholar 

  • Vanbelle, S., & Albert, A. (2009). Agreement between two independent groups of raters. Psychometrika, 74(3), 477–491. doi:10.1007/s11336-009-9116-1.

    Article  Google Scholar 

  • Zenk, S. N., Lachance, L. L., Schulz, A. J., Mentz, G., Kannan, S., & Ridella, W. (2009). Neighborhood retail food environment and fruit and vegetable intake in a multiethnic urban population. American Journal of Health Promotion, 23(4), 255–264.

    Article  Google Scholar 

Download references

Acknowledgements

Research reported in this publication was supported by the National Cancer Institute of the National Institutes of Health Under Award Number HHSN261201400034C. The authors would like to thank Dr. Phillip Dixon, a statistician in the Department of Statistics at Iowa State University, and Paul Hibbing, a Ph.D. student at the University of Tennessee for their contributions and assistance with conducting the equivalence testing analyses for the paper and for making the associated figure. The analyses provided a useful way to summarize the agreement between the novices and experts across multiple locations.

Author information

Authors and Affiliations

  1. Human Computer Interaction, Iowa State University, Ames, IA, USA

    Kelly Kalvelage, Michael C. Dorneich & Stephen Gilbert

  2. Department of Industrial and Manufacturing Systems Engineering, Iowa State University, 3004 Black Engineering Building, Ames, IA, 50011-2164, USA

    Michael C. Dorneich & Stephen Gilbert

  3. Department of Landscape Architecture, Iowa State University, Ames, IA, USA

    Christopher J. Seeger

  4. Department of Kinesiology, Iowa State University, Ames, IA, USA

    Gregory J. Welk

  5. MEI Research, Ltd, Edina, MN, USA

    Jon Moon, Imad Jafir & Phyllis Brown

Authors
  1. Kelly Kalvelage
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Michael C. Dorneich
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Christopher J. Seeger
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Gregory J. Welk
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Stephen Gilbert
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jon Moon
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Imad Jafir
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Phyllis Brown
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Michael C. Dorneich.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kalvelage, K., Dorneich, M.C., Seeger, C.J. et al. Assessing the validity of facilitated-volunteered geographic information: comparisons of expert and novice ratings. GeoJournal 83, 477–488 (2018). https://doi.org/10.1007/s10708-017-9781-z

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10708-017-9781-z

Keywords

Search

Navigation