Showing posts with label Project. Show all posts

Thursday, 12 November 2020

PyBloom coding project: Conclusions

Conclusions

This project has been quite a learning experience. I’ve had to get to grips with a lot of technologies, a lot of frameworks, and quite a learning curve. Here are some top tips:

No matter how clear the tutorials, the code never works first time. Learn by testing.
Google and YouTube are great resources for finding how to do things, but relies on being able to ask the right question. Be clear on precisely what the problem is.
Having to describe everything here has really helped reinforce the learnings.
There’s always another feature to think up and implement. But it’s important to be clear when done=done, and the program is good enough to be used.
Good enough to be used by you isn’t the same as good enough for someone else. If you want to roll out the program, have it tested by someone that doesn’t know it.

Even better if…

Over the course of this document, I’ve described how I’ve implemented the features of my program. It’s doing what I intended it to do, the lights are showing how the evenings are getting colder. It’s fine for personal consumption, but it wouldn’t be fine for others to use. Before going into new features, I should look at operationalising the code, which will be a whole different set of challenges:

The program should be available (to an acceptable service level), which means it should be deployed onto a hosted site.
I’ve done precious little formal testing. At least, operationalised code should have a testing approach consisting of: test data, test cases, expected results. At best, these tests should be completed automatically, as the code is promoted from dev to prod. This is the basis of continuous deployment.
Operational tools: manual CRUD access to the databases - because I built in a way of manually adding data, but didn’t build a way to remove it.

Thank you for joining me on this journey. Hope it’s been a little help with your own exploration. Also visit https://github.com/Schmoiger/pybloom for the full story.

Monday, 9 November 2020

Pybloom coding project part 13: Implementing version control

Implementing version control

Once you start, you can’t stop tinkering. At some point, something is going to break. So as I intend to keep this code going for a while, I implemented version control with Git. This is a feature-rich version control system, and takes care of the steps in promoting the code you’re tinkering with to code that’s ready to publish. Finally, it also integrates nicely with the GitHub service on the web where you can share your code, and pushing to remote repositories such as on my Raspberry Pi. No more cutting and pasting and associated typos.

Once completed, the setup will look like this:

Atom [UI for] -> Git on Mac <-> GitHub [also for public sharing] -> Git on RPi

Spikes

Figuring this out took a lot of searching, as I didn’t find the documentation particularly enlightening. Here’s recommended reading.

Setting up Git, GitHub and Atom

Git was already installed in my Mac, and is part of Atom by default. So no need to install anything more. But there is a lot of other setting up to be done.

Before making the first Git push, I set up the files to ignore by adding the following items to my .gitignore file. These are: the environment files that are set up by the system; or data files that are created at run time; or personal information that I don’t want you to see! Note the **/ syntax which forces the file to be ignored from all subfolders.

__pycache__* : used by Python runtime
Icon*, **/*.DS_Store : hidden files used by macOS
.git : hidden files used by Git
*_old : a useful way of hiding files you might be tinkering way
credentials.py : secrets needed for the API
**/*_bar.svg, **/*_pie.svg : these graphs are made from the data at run time
database.sqlite3 : created by the main code on first connection and updated at run time

Next up is to create the target repository on GitHub. This is done by logging into the GitHub dashboard, and creating a new repository from there. Mine is called https://github.com/Schmoiger/pybloom and I encourage you to go have a look there.

Git (the local repository) needs to know this GitHub (remote repository) URL, so the next step is to add it to the Git config. This is done from Terminal, using the following command:

git remote add origin https://github.com/<your username>/<your app name>

This command associates the name origin to the remote repository URL, which makes management within Git and Atom a little easier. (If you’re so inclined, you could instead clone my repo instead, and work on my code. We’ll do this in the next section on setting up the Raspberry Pi) Now we’ve told Git where your remote repository is, we’ve got to tell who you are, so that Git can tell GitHub. In other words, we have to set up your email address in Git to be accepted by GitHub as an authenticated user.

git config --global user.email "email@example.com"

The email address is the one set up in GitHub. It doesn’t have to be a real address; GitHub will set up a “noreply” email for you if you wish.

We don’t configure the password in the same way. Instead, if you now boot Atom you should see a login window in the GitHub pane. This asks for the login token, which you’ll need to get from https://github.atom.io/login, then paste into Atom.

Setting up the Raspberry Pi

Git comes pre-installed in Raspberry Pi OS, so no further installation necessary. But as with the Mac, there is configuration to be done.

git config --global user.name "<your username>"

git config --global user.email "<your email>"

First step is to tell the git instance on the RPi who you are. By now everyone has hundreds of username/ email combinations. Rather than creating another one, I’m re-using my GitHub identity.

git clone https://github.com/Schmoiger/pybloom.git Projects

If you’re cloning my repo into your Projects folder, use the statement above, and a copy of the repo will be cloned into a pybloom subfolder. Otherwise change the url and the destination to suit your own environment.

The final step is to copy across the credentials.py file into the pybloom folder, as it has all your secrets for the API login.

Documentation

Arguably, the most important part of the repository is the documentation. I’ve created three:

README.md - bite-sized summary of the key things you need to know to use the program
Blog post - this set of posts (not in GitHub, but on https://blog.mindrocketnow.com)
PyBloom_manual.html - all the post put together into a single document, for convenience

Putting it together

.gitignore

__pycache__*

Icon*

.git

**/*.DS_Store

**/*_bar.svg

**/*_pie.svg

*_old

credentials.py

database.sqlite3

Workflow for making changes

Make changes in dev environment, using Atom
Commit in Git (after testing), then commit to GitHub, from within Atom
SSH into the RPi, type workon pybloom to work in the virtual environment
Pull the code changes by typing git pull
Re-start the web server by typing ctrl+C then flask run --host=0.0.0.0

Friday, 6 November 2020

PyBloom coding project part 12: A little JavaScript magic

Here's an extra little something for PyBloom. In part 12 I look at how to transfer data from the Python code into the CSS of the website - not as hard as it first seems.

Colour picker utility

My stretch target was to display a table of all the colours of the Bloom lamp on a page in the website. As we've seen, the colours are in a persistent SQLite table, so all I have to do is to extract from the table, and somehow get the CSS to read the colour information. Except CSS is not an interactive language, so I needed another technology. Step forward JavaScript.

The technologies

HTML
JavaScript
Jinja2

The code

{% extends "base.html" %}

We put all the main styling into the base page, which means as before, we simply need to extend it here.

{% block app_content %}

This HTML section will be inserted into the base.html template at the position marked app_content.

We’re making use of the Bootstrap formatting for tables to make things pretty.

<tr>

<th scope="col">Temperature</th>

<th scope="col">Colour</th>

</tr>

The table consists of two columns: one for Temperature and one for the corresponding Bloom colour.

{% for row in rows %}

This is our familiar Jinja2 loop, which we want to repeat for every row in the temperature conversion lookup table.

<tr>

The first cell in the row is simply the value of temperature from the lookup table.

</tr>

The second cell is the value of the colour, a hex string. This same string is also used to identify the cell. Each cell will have its own background colour, so needs to be uniquely identified. We’ll implement this logic with a small bit of JavaScript, so let’s jump straight into it.

{% block app_js %}

As with the inserted HTML, this identifies where the custom JavaScript goes. Order is important in placing JavaScript; as this overrides the Bootstrap JavaScript, the block is after the statement that pulls Bootstrap from the CDN.

"use strict";

This is the normal preamble for JavaScript. Unlike CSS, as of HTML5 there’s no need to define type="text/javascript" as no other types are allowed. The second "use strict" command instructs the browser to use the modern (ECMAScript5 from 2009) interpreter. This is important as some commands will not work in an older interpreter, and some commands will work differently.

{% for row in rows %}

document.getElementById("{{row[2]}}").style.background = "#"+"{{row[2]}}";

We can use the same Jinja2 loop structure to set the background for each hex value cell in our table. But because each iteration of the loop creates another persistent row in the JavaScript, we need to be careful not to use variables as these would simply overwrite the previous. That’s why we have this complicated command that chains a lot of functions. Let’s pick it apart.

document.getElementById(): We assigned a unique ID for each table cell containing a hex value
"{{row[2]}}": This unique ID is the hex value string
style.background: JavaScript accesses the cell colour using this method (which is slightly different to the CSS attribute of background-color)
= "#"+"{{row[2]}}": The colour is the hex value from the lookup table, which is a string (as accepted by the SQLite database), but has to be prefixed with a hash to identify it as a hex string
; Don’t forget to finish every JavaScript statement with a semicolon - which isn’t required in Python or CSS or SQLite

The table now pulls the Hue bloom colour data from the external SQLite table and displays on a web page.

Putting it together

{% extends "base.html" %}

{% block app_content %}

<h1>Colour key</h1>

<div>

<tr>

<th scope="col">Temperature</th>

<th scope="col">Colour</th>

</tr>

{% for row in rows %}

<tr>

</tr>

{% endfor %}

</table>

</div>

{% endblock %}

{% block app_js %}

"use strict";

{% for row in rows %}

document.getElementById("{{row[2]}}").style.background = "#"+"{{row[2]}}";

{% endfor %}

</script>

{% endblock %}

This colour picker page combines a lot of technologies to achieve something that seemed quite simple, but turned out to require a bit of thought. And that's the theme for this entire project! In the next section, I'll look back at the project and draw out some conclusions. Also visit https://github.com/Schmoiger/pybloom for the full story.

Thursday, 5 November 2020

PyBloom coding project part 11: Styling the web pages

We're in the home stretch now. Here in part 11, I go through how I styled the web pages for PyBloom using a modern CSS framework.

The web pages

In the previous section, we’ve seen how to build the structure of the web app in Flask. After creating the app package, we then need to work on the HTML of the page. Finally, we’ll need to add the CSS that styles the page. To do this, the web page is written in HTML5 and CSS3, but with two scripting languages to simplify: Jinja2 for the HTML rendered by Flask, and Bootstrap CSS.

I haven’t discussed CSS yet, so let’s take a detour there. I’m a big fan of frameworks and not bothering to recreate (learn again) from first principles. The most comprehensive framework that I’ve come across for CSS is Bootstrap from Twitter. It takes care of making the page responsive, mobile-friendly with predefined styles, without the need to create the CSS from scratch. It’s a scripting language, so there is a need to learn its vocabulary which is invoked in the HTML attributes.

The structure of the HTML makes use of inheritance. There’s a base.html that declares all the common elements, such as declaring how the Bootstrap components are downloaded from the CDN, and also the navigation elements that are common for all pages. Each subsequent page extends this base.html page, to add the actual content and other custom CSS.

Technologies

HTML5, CSS3
Jinja2
Bootstrap CSS

The code

base.html - head

<!DOCTYPE html>

<head>

So far so familiar. All HTML starts with these tags.

This is the first tag required by Bootstrap.

“Bootstrap is developed mobile first, a strategy in which we optimize code for mobile devices first and then scale up components as necessary using CSS media queries. To ensure proper rendering and touch zooming for all devices, add the responsive viewport meta tag to your <head>.”

{% if title %}

<title>{{ title }} - PyBloom</title>

{% else %}

<title>Welcome to PyBloom</title>

{% endif %}

Next we have a bit of Jinja2 logic to define the title. All this is saying is: if the variable title is passed into the HTML according to the routes.py then the title is put into the metadata. If not, a default title is used. To be honest, I probably won’t invoke this logic, but it’s there if needed in the future.

These tags are needed in order to download Bootstrap from the CDN, avoiding the need to download potentially out of date components. These first <link> and <script> tags specify the version of Bootstrap CSS and Bootstrap JavaScript to be downloaded from the CDN. I’m using Bootstrap v5 which is in alpha at time of writing. I’ll clearly need to update this to the release version at some point (and hope that nothing breaks in the meantime).

I thought of defining these URLs in my rudimentary CMS, but decided against it because 1) sometimes Bootstrap classes only work with certain versions (e.g. the naming is not backwards compatible across major versions) so it’s better to reference a fixed snapshot; and 2) the complete statement is quite complex, so I can eliminate transposing errors by copying directly from the Bootstrap documentation.

{% block app_css %}{% endblock %}

This little bit of Jinja2 specifies a block where the custom CSS needed later will go.

</head>

Every website also displays a little icon in the top left of the browser tab. This last link is a combination of HTML and the very useful Jinja2 function url_for() to tell the browser where to locate this icon file. This function concatenates the absolute file path for the static folder with the filename, and inserts into the tag href attribute. We’ll use it a lot in the HTML, as it avoids hard coding file paths very nicely.

And that’s it for the head, so we close with the </head> tag.

base.html - body

The body HTML is where the Bootstrap scripting goes. The common element across the top of all of my pages will be the navigation bar. Let’s get into it!

<body>

The container model is how Bootstrap (and all CSS) describes the elements within a web page. Think of a container as a box that has a margin, border, padding, and finally the content in the centre. By declaring each container as fluid, we let the browser resize them to fill up the full screen (the viewport). Resizing, re-flowing and maybe re-factoring functionality in response to the size of the device is the essence of responsive design.

The container for the navigation bar is the navbar. In the class attribute we give it a few parameters, and Bootstrap magically does the rest of the formatting.

navbar-expand-md: the navbar will expand when the device is mid-sized; this is the breakpoint
Navbar-light: light colour scheme for the navbar elements
Bg-light: light colour scheme for the navbar background

Bootstrap provides a built-in element type of brand, with a default format that distinguishes it from other navbar elements. By making the brand element in a <span> tag, it just sits there and looks pretty. On many websites, you can click on the brand icon and go to an “about” page. To do this, just put the brand element in an anchor <a> tag with associated href attribute. I’ll use this for the other navbar elements.

PyBloom

</span>

We use the Jinja2 url_for() function in order to generate the absolute file path for the brand icon. The Bootstrap attributes of d-inline-block align-top puts the block at the top left. I like this statement as an example of how using these two scripting languages makes generating the HTML and CSS so much more readable.

This toggle button is the key element that enables my responsive design. In smaller screens, smaller than the mid-size breakpoint defined in the parent class above, the screen will not display any of the navbar elements except for the brand and this button. To display the navbar links, you just have to press this button, and the navbar elements are revealed in a column on the left. All the complexity of this design and the interactivity is summarised in these few attributes.

class="navbar-toggler": declares to Bootstrap that this button is the one that toggles the navbar in smaller screens
data-toggle="collapse": clicking the button collapses the content
data-target="navbarNavItems": links to the items in the navbar that will be collapsed/ expanded; the toggled menu bar will look for an element that has ID=navbarNavItems
Aria-expanded="false": sets the current behaviour (i.e. when page is loaded)

</button>

Because we need pretty icons everywhere, here’s an icon for the toggle button.

This class contains all the remaining navbar elements, and is the subject of the toggle button. When clicking on these elements, you’ll be taken to the web page. Bootstrap handles this subset of elements as a navbar in its own right, inheriting properties from the parent navbar.

collapse: sets the behaviour to collapse this navbar on selection
navbar-collapse: sets this navbar as the parent breakpoint for the collapse/ expand behaviour
id="navbarNavItems": connects this sub-navbar to the toggle button

This division groups all the sub-navbar elements together.

navbar-nav: these elements are of the type nav, and will be styled as such
mr-auto: the nav items after this group of elements will be pushed to the right side of the viewport, with the padding automatically calculated

<a class="nav-link" href="{{ url_for('index') }}">Weather Station</a>

</li>

Here is our first nav item, the actual element that takes us to the subsequent pages. The container for the element, and the content of element (the link) are styled differently, which is why we have two nested items. The container is simply declared to Bootstrap as a nav-item. The link itself is styled in a combination of Jinja2 and Bootstrap.

Bootstrap parameters nav-link: declares to Bootstrap as a link object, which won’t look active until the mouse hovers over it
Jinja2 script {{ url_for('index') }}: builds the absolute file path to the target page, in this case the index.html page

<a class="nav-link" href="{{ url_for('colours') }}">Colour Key</a>

</li>

The second nav item is similar to the first, except we have Jinja2 reference the colours.html file.

</li>

</div>

Finally, we have a home link on the right side of the navbar. The construction is very similar to the previous nav items, except it’s a shameless plug for my blog.

{% block app_content %}{% endblock %}

{% block app_js %}{% endblock %}

We end the base.html by provisioning a couple of slots to be extended by page-specific HTML and CSS content.

index.html

The hard work of styling each page is done in the base.html document. For each of the subsequent pages, we simply have to extend it with a small amount of additional HTML.

{% extends "base.html" %}

This Jinja2 sippet does exactly that, without needing to know the file path.

{% block app_content %}

This header tells Jinja2 where to insert the subsequent code to make up the full page code.

<h1>Latest data</h1>

<div>

<h2>Last 24 hours</h2>

As before, we use a combination of Jinja2 and Bootstrap to define and style the image elements. The Jinja2 snippet references the file that was declared in the rudimentary CMS (content.py) so that the image source points to the right graph. Bootstrap styles this as a fluid image.

</div>

{% endblock %}

After closing the division, we also need to close the block.

Putting it together

base.html

<!DOCTYPE html>

<head>

{% if title %}

<title>{{ title }} - PyBloom</title>

{% else %}

<title>Welcome to PyBloom</title>

{% endif %}

{% block app_css %}{% endblock %}

</head>

<body>

PyBloom

</span>

</button>

<a class="nav-link" href="{{ url_for('index') }}">Weather Station</a>

</li>

<a class="nav-link" href="{{ url_for('colours') }}">Colour Key</a>

</li>

</div>

</li>

</div>

</nav>

</div>

{% block app_content %}{% endblock %}

{% block app_js %}{% endblock %}

</body>

index.html

{% extends "base.html" %}

{% block app_content %}

<h1>Latest data</h1>

<div>

<h2>Last 24 hours</h2>

</div>

{% endblock %}

This is the bulk of the web page work done. I have one more web page to create, and I'll describe how in part 12. Also visit https://github.com/Schmoiger/pybloom for the full story.