Media Tech Bites: Is DVB now Irrelevant?

Here’s my short take on a piece of media technology. This week I look at a mature overripe technology, DVB. Do you agree? [blog.mindrocketnow.com]

20 years ago, when I first started out in this business, I learnt the DVB standards. This gave me a competitive advantage as a consultant; being confident with the detail down to the specification of the tables meant that I could confidently integrate broadcast systems. My last role was deeply technical as well, but the knowledge that was prized was how to use API wrappers in C++ code. Not once did I need to cast my memory back to DVB standards. For me personally, my hard-won DVB experience is now irrelevant.

DVB (and its family of standards) are still critical to broadcasters. It has fulfilled its promise of lower cost through interoperability. It’s been an objective success, and reaches millions of people daily. It’s not DVB that’s the problem, but broadcast. 

It’s becoming increasingly obvious that the future of broadcast is to be delivered over the internet. Yes, satellite has much larger footprint, yes terrestrial has fine-grained reach, but when I read the tech press, most of the arguments come from a place of wanting to continue to sweat assets that we spent so much time and effort building. None of that matters to people who consume these services. As I’ve often commented, people just want to watch TV without needing to worry about how they’re watching it.

So as broadcast becomes just another service over the internet, the skills needed to push this forward are the same as software delivery in any other industry. Just like all those pylons holding up TV transmitters around the country, my memorisation of DVB standards is now just another sunk cost.

Media Tech Bites: Blockchain’s Potential

Here’s my short take on a piece of media technology. This week I look at an immature maturing technology, blockchain. Do you agree? [blog.mindrocketnow.com]

As I’m sure you know, a blockchain is a way of recording information in a way that is (almost) impossible to cheat. It’s trivial to check, no one body (like a bank) guarantees any chain, so it’s eminently suited to industries that suffer from multiple intermediaries and opaque authorities, like finance. Or broadcast?

Blockchain is unparalleled in establishing trust across domains of ownership. Broadcast is all about passing assets across domains of ownership. If each media asset was accompanied by an immutable history, we’d know who contributed, how much, and therefore where royalties should be shared. We could see a technical history of all the corrections and transcoding over the workflow, and maybe change TV settings to compensate.

But it strikes me that blockchain isn’t the answer, because broadcast has already optimised the secured distribution of content. I don’t mean that content security is perfect because it’s far from. It’s optimised because the leakage of revenue isn’t significant enough to merit ripping out and replacing all the technology, legal and consumer investments that we’ve made.

So for broadcast, and for the time being, blockchain is still a solution looking for a problem.

PyBloom coding project: Conclusions

Conclusions

This project has been quite a learning experience. I’ve had to get to grips with a lot of technologies, a lot of frameworks, and quite a learning curve. Here are some top tips:

• No matter how clear the tutorials, the code never works first time. Learn by testing.

• Google and YouTube are great resources for finding how to do things, but relies on being able to ask the right question. Be clear on precisely what the problem is.

• Having to describe everything here has really helped reinforce the learnings. 

• There’s always another feature to think up and implement. But it’s important to be clear when done=done, and the program is good enough to be used.

• Good enough to be used by you isn’t the same as good enough for someone else. If you want to roll out the program, have it tested by someone that doesn’t know it.

Even better if…

Over the course of this document, I’ve described how I’ve implemented the features of my program. It’s doing what I intended it to do, the lights are showing how the evenings are getting colder. It’s fine for personal consumption, but it wouldn’t be fine for others to use. Before going into new features, I should look at operationalising the code, which will be a whole different set of challenges:

• The program should be available (to an acceptable service level), which means it should be deployed onto a hosted site. 

• I’ve done precious little formal testing. At least, operationalised code should have a testing approach consisting of: test data, test cases, expected results. At best, these tests should be completed automatically, as the code is promoted from dev to prod. This is the basis of continuous deployment.

• Operational tools: manual CRUD access to the databases - because I built in a way of manually adding data, but didn’t build a way to remove it.

--

Thank you for joining me on this journey. Hope it’s been a little help with your own exploration. Also visit https://github.com/Schmoiger/pybloom for the full story.

The Problem with Data Driven Decisions

This week I look at the pitfalls of being data-driven. It seems simple, and logical, so why do we end up falling back on our gut feel? [blog.mindrocketnow.com]

Should I make my next investment decision based on objective data or taking a punt based on how I feel? Seems a straightforward answer, doesn’t it - who would want to be responsible for a significant financial decision that was made on a whim? (Kinda depends if it worked out…) But making a good data-driven decision is a lot harder than just intending to do so.

What are you measuring? This simple question unravels a chain of questions that turn out to each require careful thought. Let’s pretend we’re trying to figure out whether to spend money on a cool new app feature:

• What are the business (or strategic) outcomes you’re trying to influence?

• How does the app contribute to that outcome?

• How does the feature improve the ability of the app to contribute to that outcome?

• What metric can we put on that improvement?

• How can we measure those metrics? 

• Are those measurements allowed in our privacy guidelines?

• What would be the impact on those quantities if we didn’t build the app?

• Is there anything else we could do, at less cost, that would improve that metric, even in part?

• Is there anything else we could do, at same cost, that would improve an alternative, more important metric?

Most of the time, we don’t have a good answer to all of those questions, oftentimes because to do so thoroughly will take a lot of effort, more than the effort to ship the feature itself. So we end up making a best guess - aka taking a punt.

I’m really attracted to the concept of lean development, which codified learning by doing. The idea is to ship features as frequently as possible, measure impact upon metrics, and improve or discard depending on whether it’s a positive or negative impact. By keeping this feedback loop really short, we risk less wasted development effort. As a side-effect, we maintain focus on the metrics that matter to us, and maintain a cadence of shipping features.

As before, it depends upon the metric. If we optimise to a vanity metric (one that doesn’t align with a business objective) or a proxy metric (one that doesn’t align well with a business objective), then we might miss the business objective. We might optimise for a very fast playback start but miss the fact that consumers are much more worried about not finding programmes that they like.

After all that, you might be thinking that I advise avoiding making gut feel decisions. You’d be right - gut feel is basically your thought heuristics kicking in, reinforcing all your unconscious biases. Taking a moment to consider rather than react is a good rule. But that doesn’t mean you shouldn’t use your feelings. I’m also a strong believer in eating your own cooking. 

We should be building apps for people, and we need to understand people in detail if the app is going to make a difference to them. The person you get to see the most is yourself, so you should be able to analyse your own reactions to your app in the best detail.

Be data driven but guided by context. The best context is you.

Media Tech Bites: Importance of Metadata

Here’s my short take on a piece of media technology. This week I look at why metadata is the most important part of media tech. Do you agree? [blog.mindrocketnow.com]

Broadcast is in the middle of a technology revolution. 20 years ago, I was cabling routers to encoders, making sure that the labels were correct. Now I’m working with engineers to ensure the virtual routing through our AWS services is working as it should, and feeding iOS and Android apps properly.

Not everything is changing. Then and now, we use metadata to describe the services being delivered. In the world of DVB then, the metadata was in the main the System Information tables that told the Set Top Boxes where to find the broadcast services in the frequency spectrum. Now, as then, the metadata drives content search and discovery. However now, unlike then, metadata drives a far richer discovery experience; we can see trailers where we used to see thumbnails, we can link to IMDB articles where before we had a character-limited synopsis, and we can select the next programme based on what we’ve just watched or even what mood we’re in.

More fundamentally, metadata is starting to drive how the services are created. The cabling and labels are abstracted from the creation of services by software drivers. Middleware orchestrates those drivers into services. Those services can be orchestrated together in very visual “low code” ways. They can be defined in terms of metadata alone. This means UI and UX designers can put together interesting app experiences without needing to know Kotlin or Swift. It will mean that experiences can be put together programmatically, such as a UI that is reactive to the content that you’re watching, or the context that you’re watching it in.

If this all seems familiar, it’s because the enterprise IT sector went through this change last decade. It’s experience is that metadata drives the creation of the service, the consumption of the service, and the quality assurance of the service. To stay relevant, it’ll be necessary for all of us to be metadata-literate.

Pybloom coding project part 13: Implementing version control

 

Implementing version control

Once you start, you can’t stop tinkering. At some point, something is going to break. So as I intend to keep this code going for a while, I implemented version control with Git. This is a feature-rich version control system, and takes care of the steps in promoting the code you’re tinkering with to code that’s ready to publish. Finally, it also integrates nicely with the GitHub service on the web where you can share your code, and pushing to remote repositories such as on my Raspberry Pi. No more cutting and pasting and associated typos.

Once completed, the setup will look like this:

Atom [UI for] -> Git on Mac <-> GitHub [also for public sharing] -> Git on RPi

Spikes

Figuring this out took a lot of searching, as I didn’t find the documentation particularly enlightening. Here’s recommended reading.

• Git and Atom and GitHub

• About remote repositories 

• Getting started with Git and Raspberry Pi 

Setting up Git, GitHub and Atom

Git was already installed in my Mac, and is part of Atom by default. So no need to install anything more. But there is a lot of other setting up to be done. 

Before making the first Git push, I set up the files to ignore by adding the following items to my .gitignore file. These are: the environment files that are set up by the system; or data files that are created at run time; or personal information that I don’t want you to see! Note the **/ syntax which forces the file to be ignored from all subfolders.

• __pycache__* : used by Python runtime 

• Icon*, **/*.DS_Store : hidden files used by macOS

• .git : hidden files used by Git

• *_old : a useful way of hiding files you might be tinkering way 

• credentials.py : secrets needed for the API

• **/*_bar.svg, **/*_pie.svg : these graphs are made from the data at run time

• database.sqlite3 : created by the main code on first connection and updated at run time

Next up is to create the target repository on GitHub. This is done by logging into the GitHub dashboard, and creating a new repository from there. Mine is called https://github.com/Schmoiger/pybloom and I encourage you to go have a look there.

Git (the local repository) needs to know this GitHub (remote repository) URL, so the next step is to add it to the Git config. This is done from Terminal, using the following command:

git remote add origin https://github.com/<your username>/<your app name>

This command associates the name origin to the remote repository URL, which makes management within Git and Atom a little easier. (If you’re so inclined, you could instead clone my repo instead, and work on my code. We’ll do this in the next section on setting up the Raspberry Pi) Now we’ve told Git where your remote repository is, we’ve got to tell who you are, so that Git can tell GitHub. In other words, we have to set up your email address in Git to be accepted by GitHub as an authenticated user.

git config --global user.email "email@example.com"

The email address is the one set up in GitHub. It doesn’t have to be a real address; GitHub will set up a “noreply” email for you if you wish.

We don’t configure the password in the same way. Instead, if you now boot Atom you should see a login window in the GitHub pane. This asks for the login token, which you’ll need to get from https://github.atom.io/login, then paste into Atom.

Setting up the Raspberry Pi

Git comes pre-installed in Raspberry Pi OS, so no further installation necessary. But as with the Mac, there is configuration to be done.

git config --global user.name "<your username>"

git config --global user.email "<your email>"

First step is to tell the git instance on the RPi who you are. By now everyone has hundreds of username/ email combinations. Rather than creating another one, I’m re-using my GitHub identity.

git clone https://github.com/Schmoiger/pybloom.git Projects

If you’re cloning my repo into your Projects folder, use the statement above, and a copy of the repo will be cloned into a pybloom subfolder. Otherwise change the url and the destination to suit your own environment.

The final step is to copy across the credentials.py file into the pybloom folder, as it has all your secrets for the API login.

Documentation

Arguably, the most important part of the repository is the documentation. I’ve created three:

• README.md - bite-sized summary of the key things you need to know to use the program

• Blog post - this set of posts (not in GitHub, but on https://blog.mindrocketnow.com)

• PyBloom_manual.html - all the post put together into a single document, for convenience

Putting it together

.gitignore

__pycache__*

Icon*

.git

**/*.DS_Store

**/*_bar.svg

**/*_pie.svg

*_old

credentials.py

database.sqlite3

Workflow for making changes

1. Make changes in dev environment, using Atom

2. Commit in Git (after testing), then commit to GitHub, from within Atom

3. SSH into the RPi, type workon pybloom to work in the virtual environment

4. Pull the code changes by typing git pull

5. Re-start the web server by typing ctrl+C then flask run --host=0.0.0.0

PyBloom coding project part 12: A little JavaScript magic

Here's an extra little something for PyBloom. In part 12 I look at how to transfer data from the Python code into the CSS of the website - not as hard as it first seems.

Colour picker utility

My stretch target was to display a table of all the colours of the Bloom lamp on a page in the website. As we've seen, the colours are in a persistent SQLite table, so all I have to do is to extract from the table, and somehow get the CSS to read the colour information. Except CSS is not an interactive language, so I needed another technology. Step forward JavaScript.

The technologies

• HTML

• JavaScript

• Jinja2

The code

{% extends "base.html" %}

We put all the main styling into the base page, which means as before, we simply need to extend it here.

{% block app_content %}

This HTML section will be inserted into the base.html template at the position marked app_content.

<table class="table">

We’re making use of the Bootstrap formatting for tables to make things pretty. 

<tr>

  <th scope="col">Temperature</th>

  <th scope="col">Colour</th>

</tr>

The table consists of two columns: one for Temperature and one for the corresponding Bloom colour.

{% for row in rows %}

This is our familiar Jinja2 loop, which we want to repeat for every row in the temperature conversion lookup table.

<tr>

  <td>{{row[1]}}</td>

The first cell in the row is simply the value of temperature from the lookup table.

  <td id="{{row[2]}}">{{row[2]}}</td>

</tr>

The second cell is the value of the colour, a hex string. This same string is also used to identify the cell. Each cell will have its own background colour, so needs to be uniquely identified. We’ll implement this logic with a small bit of JavaScript, so let’s jump straight into it.

{% block app_js %}

As with the inserted HTML, this identifies where the custom JavaScript goes. Order is important in placing JavaScript; as this overrides the Bootstrap JavaScript, the block is after the statement that pulls Bootstrap from the CDN.

<script>

  "use strict";

This is the normal preamble for JavaScript. Unlike CSS, as of HTML5 there’s no need to define type="text/javascript" as no other types are allowed. The second "use strict" command instructs the browser to use the modern (ECMAScript5 from 2009) interpreter. This is important as some commands will not work in an older interpreter, and some commands will work differently. 

{% for row in rows %}

  document.getElementById("{{row[2]}}").style.background = "#"+"{{row[2]}}";

We can use the same Jinja2 loop structure to set the background for each hex value cell in our table. But because each iteration of the loop creates another persistent row in the JavaScript, we need to be careful not to use variables as these would simply overwrite the previous. That’s why we have this complicated command that chains a lot of functions. Let’s pick it apart.

• document.getElementById(): We assigned a unique ID for each table cell containing a hex value

• "{{row[2]}}": This unique ID is the hex value string

• style.background: JavaScript accesses the cell colour using this method (which is slightly different to the CSS attribute of background-color)

• = "#"+"{{row[2]}}": The colour is the hex value from the lookup table, which is a string (as accepted by the SQLite database), but has to be prefixed with a hash to identify it as a hex string

• ; Don’t forget to finish every JavaScript statement with a semicolon - which isn’t required in Python or CSS or SQLite

The table now pulls the Hue bloom colour data from the external SQLite table and displays on a web page.

Putting it together

{% extends "base.html" %}

{% block app_content %}

  <h1>Colour key</h1>

  <div>

    <table class="table">

      <tr>

        <th scope="col">Temperature</th>

        <th scope="col">Colour</th>

      </tr>

      {% for row in rows %}

      <tr>

        <td>{{row[1]}}</td>

        <td id="{{row[2]}}">{{row[2]}}</td>

      </tr>

      {% endfor %}

    </table>

  </div>

{% endblock %}

{% block app_js %}

  <script>

    "use strict";

    {% for row in rows %}

      document.getElementById("{{row[2]}}").style.background = "#"+"{{row[2]}}";

    {% endfor %}

  </script>

{% endblock %}

This colour picker page combines a lot of technologies to achieve something that seemed quite simple, but turned out to require a bit of thought. And that's the theme for this entire project! In the next section, I'll look back at the project and draw out some conclusions. Also visit https://github.com/Schmoiger/pybloom for the full story.