### Learning to do stuff poorly w/ @mstafford; #### AKA, "Did I just get *worse* at this?" I'm currently working on my steem (d)App* project a bit more frequently as the weather cools down, and I just got remotivated to work on a few things that I had been putting on the back-burner. Specifically, I've been meaning to leverage the powerful [GeoDjango](https://docs.djangoproject.com/en/2.2/ref/contrib/gis/) package: _{**I'd be lying if I said it was really a (d)ecentralized application*} ![image.png](https://files.steempeak.com/file/steempeak/mstafford/l1NQSOLJ-image.png) The earlier version of EXHAUST actually had this package implemented, but I hadn't really done anything with it yet save some testing. In tandem, I'm going to document my process here in an effort to do just that -- document. I'm horrible at writing my process and assumptions down, and I need to change. I barely even comment anything when I spray Python everywhere, and it takes me forever to remember WTF I was thinking. Today -- it all changes. Sorta... People that know me have told me I'm pretty casual. I'm gonna keep it this way, and just kinda stream-of-conciousness this guide. For some of the basics w/ Django -- I recommend checking out some of the goods of @steempytutorials -- like [this one on starting a Steem Django app](https://steemit.com/utopian-io/@steempytutorials/part-0-create-steem-web-applications-with-django-and-steem-python). So w/ that in mind, lets dig into it. *** Before I start this journey, it's probably a pretty useful exercise to figure out *WHAT* we want to accomplish before we just start go fuckin' crushing it. So with that in mind, here are some objectives: ### Objectives: 1. Install GeoDjango and dependencies; 2. Set up a new Postgres database w/ appropriate 'extensions' and get our Django project to interact w/ new database; a) This requires implementing some database routing, since we're going to leave our original database in operation; 3. Figure out how to *display* the map object; 4. Figure out how to display *content from our new GIS Database* in maps that can be viewed by anyone; a) The GeoDjango tutorials are actually quite lacking in this aspect -- so we'll find something else / get creative here; 5. Think of what we can do to make this even better; *** ### Assumptions: As mentioned above, I want to set up some GIS functionality for my project that's currently process. With that in mind, I'm not going to go into detail on how to install Django itself -- there are some *awesome* official documentation and resources for this elsewhere. So, I'm starting this guide with the following assumptions: * We're using ***Ubuntu for an OS***; * We ***already have a working installation of Django running***; * We're using ***Postgres as our backend Database Manager -- and have already installed it***; * Fun fact -- my Django project is hosted by [DigitalOcean](https://www.digitalocean.com/) and they provide a really thorough guide on [How to Set Up Django w/ Postgres and Nginx](https://www.digitalocean.com/community/tutorials/how-to-set-up-django-with-postgres-nginx-and-gunicorn-on-ubuntu-16-04) |Django|Postgres|Nginx|Ubuntu| |---|---|---|---| |![DjangoLogo](https://www.djangoproject.com/m/img/logos/django-logo-positive.png)|![PostgresLogo](https://wiki.postgresql.org/images/3/30/PostgreSQL_logo.3colors.120x120.png)|![NginxLogo](https://www.nginx.com/wp-content/uploads/2018/08/NGINX-logo-rgb-large.png)|![UbuntuLogo](https://assets.ubuntu.com/v1/8dd99b80-ubuntu-logo14.png)| *** ## Step 1 - Install GeoDjango and Dependencies: This section will be enjoyably brief, as the instructions on the GeoDjango [installation instructions](https://docs.djangoproject.com/en/2.2/ref/contrib/gis/install/) were concise, up to date, and easy. 1. First -- lets start up a new django-app called 'webmap': ```python manage.py startapp webmap``` 2. Install GeoDjango: a. hahaha! Tricked you sucka! This shit comes preloaded w/ Django, and you just gotta turn it on. It's like you're having your *true powers unlocked* like that episode of Dragon Ball Z. ![Dragon Dragon Ball....Drago Ball Zeeeeeeeeeeeeee.....](https://media.giphy.com/media/1kTRympxgTMqo1UrLq/giphy.gif) So. It's already installed. We just gotta enable it in our project settings. Go to your favorite editor (I'm too dumb to learn ```vim```, so I tend to use ```nano```), and edit your DjanoProjects settings file: ```/path/to/project$ nano myproject/settings.py``` and make sure you have the GIS contrib package, and our new ```webmap``` app 'installed' to your project: ``` INSTALLED_APPS = [ ..., 'django.contrib.auth', 'django.contrib.gis', 'webmap', ..., ] ``` I'm also showing that I have the ```django.contrib.auth``` package installed, as it might come up later. My project obviously allows users to login and manage various things on their account -- and I'd like to be able to link ```Users``` from the ```auth``` package, to shapes and geometries in the ```gis``` package. ```Auth``` is automatically added whenever you start a new Django project, I believe, and most of the time there's probably no reason to remove it. 3. Install GeoSpatial Libraries and other dependencies; a. [binutls](https://packages.ubuntu.com/xenial/binutils) -- not gonna lie, I'm not 100% certain what this is. A quick google search says it's a **GNU assembler, linker and binary utilities**. Sounds like low-level computer wizardry sorta stuff -- and that's a good enough explanation for me. Let me know in the comments if you've got a laymans description of what this is. b. [libproj](https://packages.ubuntu.com/xenial/libproj-dev) -- Maps have all sorts of different projection types. That's why on some maps, various countries look huge -- while on others, they look less huge. This library is a **Cartographic projection library (development files)** . More details on this sorta stuff can be found [here](https://en.wikipedia.org/wiki/Map_projection) c. [gdal](https://packages.ubuntu.com/xenial/gdal-bin) -- GDAL is a translator library for raster and vector geospatial data formats. More info on GDAL can be found [here](https://gdal.org/) Since I'm doing all of this on a server w/ an Ubuntu OS, I can install all of these packages w/ a quick ```sudo apt-get install binutils libproj-dev gdal-bin```. ![Office Space](https://media.giphy.com/media/zcCGBRQshGdt6/giphy.gif) Gotta love it when shit is easy. ## Step 2 - Create a new PostGIS spatial database, and get Django working with *multiple databases*: So I've never really done it, but I would *IMAGINE* that you could be totally fine with operating on a single database. It's probably even easier. But I'm not proud of myself for always doing things the easy way. I typically trudge through things in a ridiculous order, and make my projects needlessly harder than they have to be. So I'm approaching things like you already have a fairly stable project, and you just thought of something cool you could do if you *just had some webmaps*. We're gonna add a new database, and then operate on both *at the same time*. ![two at the same time](https://media.giphy.com/media/UoXRM5f65wJxxzk2rK/giphy.gif) #### Lets start a new Postgres Database Git to yer terminal! ``` postgres@myserver:/path/to/project$ createdb geodjangodb postgres@myserver:/path/to/project$ psql geodjangodb psql (10.10 (Ubuntu 10.10-0ubuntu0.18.04.1)) Type "help" for help. geodjangodb=# CREATE EXTENSION postgis; CREATE EXTENSION geodjangodb=# \q ``` Dope! Database is created and ready to be populated with our sweet sweet geometry. We already "installed" the ```django.contrib.gis```in our setup earlier. Lets jump back into ```myproject/settings.py``` file and define our second (or first, depending how far ahead you plan your life). Scroll down until you get to your ```DATABASE``` definition. You'll see something like this: ``` # Database # https://docs.djangoproject.com/en/2.2/ref/settings/#databases DATABASES = { 'default': { 'ENGINE': 'django.db.backends.postgresql_psycopg2', 'NAME': 'projectdbname', 'USER': 'dbadmin', 'PASSWORD': 'S3CR3TP$SSW0RD', 'HOST': 'localhost', 'PORT': '', }, } ``` lets add a second entry into the ```DATABASES``` dictionary, and lets tell django to call this database ```maptest```: ``` # Database # https://docs.djangoproject.com/en/2.2/ref/settings/#databases DATABASES = { 'default': { 'ENGINE': 'django.db.backends.postgresql_psycopg2', 'NAME': 'projectdbname', 'USER': 'dbadmin', 'PASSWORD': 'S3CR3TP$SSW0RD', 'HOST': 'localhost', 'PORT': '', }, 'maptest': { 'ENGINE': 'django.contrib.gis.db.backends.postgis', 'NAME': 'geodjangodb', 'USER': 'dbadmin', 'PASSWORD': 'S3CR3TP$SSW0RD', 'HOST': 'localhost', 'PORT': '', } } ``` We're getting pretty close to ready for migrating our project and getting everything ready -- but we've got two more things to do: * Create a model for our Database and use some actual spatial model fields; and * We need to set up some database routing, so our spatial-data entries *only get writtien to the database that supports it*. This is called ```routing```... probably... Lets edit our ```webmap/models.py``` file -- and because it's probably useful, lets create the same ```WorldBorders``` that's covered in the GeoDjango [tutorial](https://docs.djangoproject.com/en/2.2/ref/contrib/gis/tutorial/#defining-a-geographic-model) with a super efficienct (read: shameless) copy/paste: ``` from django.contrib.gis.db import models class WorldBorder(models.Model): # Regular Django fields corresponding to the attributes in the # world borders shapefile. name = models.CharField(max_length=50) area = models.IntegerField() pop2005 = models.IntegerField('Population 2005') fips = models.CharField('FIPS Code', max_length=2) iso2 = models.CharField('2 Digit ISO', max_length=2) iso3 = models.CharField('3 Digit ISO', max_length=3) un = models.IntegerField('United Nations Code') region = models.IntegerField('Region Code') subregion = models.IntegerField('Sub-Region Code') lon = models.FloatField() lat = models.FloatField() # GeoDjango-specific: a geometry field (MultiPolygonField) mpoly = models.MultiPolygonField() # Returns the string representation of the model. def __str__(self): return self.name ``` The important thing to take away from the above copypaste is the ```mpoly``` field near the bottom. Notice that ```from django.contrib.gis.db``` we imported our ```models``` module. From this, we get the new ```MultiPolygonField``` ```class```. Start a new ```python``` file, ```routers.py```, in your ```/path/to/project/myproject/```folder. Same place where we found ```settings.py``` and blindly chuck this shit in there: ``` class DBRouter: """ A router to control all database operations on models in the world application. """ def db_for_read(self, model, **hints): """ Attempts to read webmap models go to geodjangodb. """ if model._meta.app_label == 'webmap': return 'maptest' return None def db_for_write(self, model, **hints): """ Attempts to write webmap models go to geodjangodb. """ if model._meta.app_label == 'webmap': return 'maptest' return None def allow_migrate(self, db, app_label, model_name=None, **hints): """ Make sure the webmap app only appears in the 'geodjangodb' database. """ if app_label == 'webmap': return db == 'maptest' return None ``` Cool! So what this does, is override a few standard functions in django in terms of how it interacts with its backend database. It will, by default, go to the database you've set as "default" in teh ```settings.py``` file. Otherwise, for reading from, or writing to the database, it'll look for the ```maptest``` database if we're dealing with our new ```webmap``` app. Also, since the geometry fields in our spatial database require whatever kinda neat extension stuff we did before, we need to make sure that the ```webmap``` app only appears in the ```maptest``` database. Lets do it. The moment we've been waiting for... Lets make like a flock of birds and ```migrate```! Note that if you just run the standard ```python manage.py migrate``` you'll ***ONLY*** run the migration operation for the ***DEFAULT*** database. In order to migrate for our new ```maptest``` database, we just need to specify ```python manage.py migrate --database=maptest``` ``` (env) me@server:/path/to/project$ python manage.py makemigrations sanity check! Migrations for 'webmap': webmap/migrations/0001_initial.py - Create model WorldBorder (env) me@server:/path/to/project$ python manage.py migrate --database=maptest sanity check! Operations to perform: Apply all migrations: existingapp1, admin, auth, existingapp2, contenttypes, steemkeychain, webmap, existingapp3, existingapp4, sessions Running migrations: Applying contenttypes.0001_initial... OK Applying auth.0001_initial... OK Applying A BUNCH MORE OF THESE... ``` Okay. Not gonna lie. That felt like more work than I was led to believe this was going to be. However -- things *seem* to be functioning, and I'm willing to ignorantly believe that it's going to get better.. *This is a cheeky, hidden, mid-sentence post to see if anyone has read this far into the post -- the first account to comment that "@exhaust is the best damn GIS-enabled, activity-based, STEEM project around!" gets 10 STEEM!* It'll all be worth it when we develop our cream of the crop GIS enhanced steem-app. Hooooo boy. ![creamy](https://media.giphy.com/media/fU8EfA0FM90qs/giphy.gif) ## Step 3 - Get some maps showing up on our website: So one of the things we did in the [previous section]() was to define our ```WorldBorder``` model. We defined this to work nicely w/ the ```*.shp``` (shapefile) that's provided in the [GeoDjango tutorial](https://docs.djangoproject.com/en/2.2/ref/contrib/gis/tutorial/#importing-spatial-data). It contains borders for the countries of the world, and is useful in performing spatial searches by country. After you've loaded the data (you should have defined a python file for ```webmap/load.py``` and then executed the ```webmap.load.run()``` method). This should have created an entry in our ```WorldBorder``` model for every country that's defined in the shapefile. The quickest way to see this data on *a map*, is to register our ```webmap``` app for the django ```admin``` site -- this is covered, again, in the [GeoDjango Tutorial](https://docs.djangoproject.com/en/2.2/ref/contrib/gis/tutorial/#putting-your-data-on-the-map) but I'll copy/paste for now, and then we'll take it a step further: Register ```WorldBorder``` in the ```webmap/admin.py```: ``` ### webmap/admin.py from django.contrib.gis import admin from .models import WorldBorder admin.site.register(WorldBorder, admin.GeoModelAdmin) ``` and then ensure your admin site is pathed, and also import the ```admin``` module from ```django.contrib.gis```: ``` from django.contrib.gis import admin from django.urls import include, path urlpatterns = [ path('admin/', admin.site.urls), url(r'^maptest/', include('webmap.urls')), ] ``` Since we *KNOW* we're going to have to do this eventually anyways, we've added the entry for our new ```webmap``` app ```urls.py``` file. So if we navigate a web browser to ourproject.site/maptest/ -- we'll be directed to the urls listed in our ```webmap``` app -- which we'll define shortly. Now, we can see all the country borders we just imported, if we go to our admin page at https://localhost:8000/admin : ![image.png](https://steemitimages.com/0x200/https://files.steempeak.com/file/steempeak/mstafford/dTQAsJBX-image.png) | ![image.png](https://steemitimages.com/0x200/https://files.steempeak.com/file/steempeak/mstafford/0jDJveVU-image.png) ...Cool? Sooo.. People have to log in to the *ADMIN* page to see stuff? That seems dumb... #### and it IS dumb. ## Step 4 - Link content from our fancy-pants PostGIS geospatial database into some Django views / forms, and HTML templates: *(pro-top: if you use big words / phrases like "geospatial database", people generally just assume you know what you're talking about, even if you don't).* So lets make a couple quick and dirty placeholders just to get things working on a site that's not the admin site. We're going to write a new ```view```, a new ```form```, a very simple ```html``` file, and add a couple lines to our ```urls.py``` files. It's gonna be fun. Hopefully. #### The form! ``` ## webmap/forms.py from django.contrib.gis import forms class GeoForm(forms.Form): label="Maps!" mpoly = forms.MultiPolygonField(widget= forms.OSMWidget(attrs={ 'map_width': 800, 'map_height': 500 }) ) ``` A couple notes on the above: * We've imported the forms module for the ```django.contrib.gis``` package; * We're using the OSM (OpenStreetMap) widget to display our maps, rather than Google or OpenLayers. * Our form class is called ```GeoForm```, and our default map size is 800 x 500 pixels #### The view! ``` ### webmap/views.py from webmap.forms import GeoForm # Generic Map Areas from webmap.models import WorldBorder from django.http import HttpResponse, Http404, HttpResponseRedirect from django.contrib.gis.geos import MultiPolygon from django.template import loader from django.views import View import re # Create your views here. def world(request): countries = WorldBorder.objects.filter(pop2005__gte=0).order_by('name') all_countries = 'SRID=4326;MULTIPOLYGON (' for idx, border in enumerate(countries): temp_text = re.sub('SRID=4326;MULTIPOLYGON $','',border.mpoly.ewkt) temp_text = re.sub('$\)\)','))',temp_text) if idx > 0: all_countries += "," all_countries += temp_text all_countries += ")" all_countries = MultiPolygon().from_ewkt(all_countries) geo = GeoForm() geo.fields['mpoly'].initial = all_countries context = { 'map':geo, 'countries': countries, } template = loader.get_template('maptest/mapbasic.html') return HttpResponse(template.render(context,request)) def country(request, country_id): countries = WorldBorder.objects.filter(pop2005__gte=0).order_by('name') selected = WorldBorder.objects.get(pk=country_id) geo = GeoForm() geo.fields['mpoly'].initial = selected.mpoly context = { 'map':geo, 'countries': countries, 'selected': country, } template = loader.get_template('maptest/mapbasic.html') return HttpResponse(template.render(context,request)) ``` A couple notes on the above: * We're importing our ```GeoForm``` from the ```forms.py``` we just created; * We're importing our ```WorldBorder``` model from the ```models.py``` file created much earlier; * Importing a couple standard django http response modules that I just kinda copy-paste into new view files; * We're also importing the ```MultiPolygon``` class from ```django.contrib.gis.geos```; * This is going to let us define new geometry in our map -- our "new" geometry will be a "multipolygon" of all the countries in our "WorldBorders" data put together; * ```countries``` is a queryset of all countries organized by date; * ```all_countries``` is a variable for our new geometry. We're adding some text at the start to define the **projection** (SRID=4326), and **geometry type** (MULTIPOLYGON); then * iterating through all the countries to: * remove that same text from each line for the specific country; * add the geometry definition for that specific country as a new "polygon" in our "multipolygon" geometry held in ```all_countries``` * Create a new ```MultiPolygon``` class object from the ```all_countries``` data, and use it to define the initial data in our ```GeoForm```; * Then we package up our ```GeoForm``` and list of ```countries``` in some context variables to send to our template! * We've also created a second view that will handle displaying an *INDIVIDUAL* country from our data set. Managed in a similar fashion -- but we don't have to bundle a bunch of shit together. Getting close now! #### The URLs! ``` ### webmap/urls.py from django.urls import include, path from . import views urlpatterns = [ path("", views.world, name="world"), path("c/",views.country, name='maps'), ] ``` Some notes on the above: * Make sure you import the ```include``` and ```path``` modules; * The first entry takes us to our bastardized "all countries" display. This is directed to the ```webmap.views.world``` method we created; * The second entry will take a variable from the hyperlink for which country to show, which is directed to the ```webmap.views.country``` method we definte above. #### The template! ``` ### webmap/templates/mapbasic.html

Change Country

{% for country in countries %}

{% endfor %}

Browsing Maps for {{ selected.name }}

Choose a country on the left!

``` A couple notes on the above: * We're importing [Bootstrap 4](https://getbootstrap.com/docs/4.0/getting-started/introduction/) CSS and JS for our simple layout; * We iterate through all the countries in our database, and create a link in the sidebar on the left of our page; * We call up a ```{% csrf_token %}``` to prevent our users getting played like a bunch of chumps. Protect [yo-self](https://portswigger.net/web-security/csrf/tokens); * We're displaying our prepopulated map ![World Map](https://files.steempeak.com/file/steempeak/mstafford/eDle91sI-image.png) ![Canada Only](https://files.steempeak.com/file/steempeak/mstafford/QRKLIt0D-image.png) *Adding a bit more detail can go a long way* ## Step 5 - Think about some cool stuff we could do to make this even better -- or at least, less bad: So I was going to continue this rant / tutorial / evidence-of-my-incompetence a bit more -- but I think this is definitely long enough. I'll finish things off with a few bullet points of what I'm thinking would be a great way to expand / polish this idea. So without further ado, here are some of my thoughts: * Utilize the [Django-Leaflet](https://github.com/makinacorpus/django-leaflet) project that others have worked on. This should improve the UX with maps; * I'm hoping / assuming that "hyperlink clickable geometries" is something that Leaflet can achieve. * Implement some controls so (specific) users are able to edit/create/manage geometries; * Put in a bit more detail, and grab provinces/states/territories; * You can see from the image above that I've already done this on my project; * Allow users to query their data by geo area. * Compile and save custom figures; * Extend this and link w/ SMT / SCOT tokens so business "register" their geo-location based on said SMT / SCOT tokens; * Extend so project users can link their profile to areas that their responsible for managing; There's all sorts of different ways that information can be broken down and/or categorized based one geographic zones. The sky is essentialy the limit. I've got some pretty interesting plans for GIS and [EXHAUST](https://xhaust.me)! I hope you join me as we explore this path together moving forwards. *** What do you think are some cool use cases for coupling STEEM / SCOT/ SMTs and GIS / spatial analysis? YOu've seen all these #team-canada or #team-malaysia tags floating around. Maybe this could all just be in the background, and there would be some group metrics / rewards just for being in a certain part of the world. Or maybe this would be a great way to meet new Steemians in your area. Or maybe you could sort @travelfeed curated content by areas on a map, and see reviews for different cities / countries on a map. I'm keen to hear what you think. Let me know!